The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
HellaSwag: Can a Machine Really Finish Your Sentence?
Open accessAbstract
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys."With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached.Does this mean that machines can perform human level commonsense inference?In this paper, we show that commonsense inference still proves di cult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset.Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle ( †48%).We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers.AF proves to be surprisingly robust.The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.Our construction of HellaSwag, and its resulting di culty, sheds light on the inner workings of deep pretrained models.More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is no 'ground-truth' implementation from the original benchmark authors, and many popular benchmarks (Paperno et al., 2016; Clark et al., 2018; Zellers et al., 2019) that have been 'retrofitted' for compatibility with autoregressive LMs.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zellers et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 1 biasing the comparison to human performance.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Traditional language model benchmarks, such as HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), MMLU (Hendrycks et al., 2021), GSM-8K (Cobbe et al., 2021), and BIG-Bench (Srivastava et al., 2023), primarily assess LLMs on tasks like commonsense reasoning and standardized exams.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“for evaluating the following tasks that measure common sense reasoning: LAMBADA (OpenAI version in LM Evaluation Harness) (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge, ARC-easy (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2021).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Zellers et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Self-citationDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Research works printed but not linked: 2
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for e cient text classification. In Proceedings of the 15th Confer- ence of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 427-431.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Cited by
Sorted by citations
Items 701–800 of 908
Training Dynamics of a 1.7B LLaMa Model: A Data-Efficient Approach
Li, M. Q., Fung, B. C. M., & Huang, S.-C. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228044
“An intriguing phenomenon we observed is the model’s sensitivity to a small change in the prompt formatting.”
FedSIT: Efficient Federated Fine-Tuning with Model Splitting and Importance-Based Tuning
Gu, X., Sun, X., Feng, C., Wang, X., & Chen, X. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227876
Younis, H. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5630470
ToMamba: Towards Token-Efficient Mamba Architecture on FPGA
Shi, K., Cao, Y., Du, Y., Chen, J., Yu, J., & Wang, K. (2025) · doi:10.1109/iccad66269.2025.11240811
Li, Z., Wen, H., Hu, W., Luo, Z., Chen, L., Wang, S., & Wu, S. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3346-6_1
Position Bias Across LLM Model Families
Vella, S., Sharieh, S., & Ferworn, A. (2025) · Interdisciplinary Conference on Electrics and Computer (INTCEC) · doi:10.1109/intcec65580.2025.11255828
“HellaSWAG [17] is a more challenging version which uses grounded truth from WikiHow and ActivityNet instead of LLM-generated incorrect answers.”
WizardEvent: Empowering Event Reasoning by Hybrid Event-Aware Data Synthesizing
Tao, Z., Chen, X., Jin, Z., Bai, X., Zhao, H., Hu, W., Tao, C., & Ma, S. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3634839
“We also compare to general test datasets: factual understanding (CSQA [34], HELLASWAG [35], OBQA [36]) and logical reasoning(ARC-Challenge [37], Winogrande [38], RACE-middle, and RACE-high [39]).”
POU: Outlier Uniformity-Based Layer-Wise Pruning Method for Large Language Models
Ma, Z., Cui, Y., & Qin, Y. (2025) · IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA) · doi:10.1109/ispa67752.2025.00140
“We use two metrics: language modeling, measured by WikiText perplexity [8], and zero-shot accuracy on seven common-sense benchmarks: BoolQ [7], RTE [6], HellaSwag [5], WinoGrande [4], ARC (Easy, Challenge) [3], and Open-bookQA [2].”
Narayanaswami, S. K., Srinivasan, G., & Ravindran, B. (2025) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped65674.2025.11261772
“3) Natural Language Processing: We evaluate the GPT and OPT models on the Hellaswag [17] benchmark (validation split) with 10,042 multiple-choice reasoning questions.”
Commonsense Reasoning 2.0: Symbolic Approaches to Close the GPT-4 Gap
Gumelar, A. B., Alamsyah, N., Adi, D. P., Winardi, S., & Sooai, A. G. (2025) · International Conference on Information Technology and Computing (ICITCOM) · doi:10.1109/icitcom66635.2025.11264873
DEFT: Data-Efficient Fine-Tuning Through Multi-Dimensional Data Selection
Dai, S., Liu, X., & Yu, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3642562
A Benchmark to Evaluate LLMs’ Proficiency on Italian Student Competencies
Mercorio, F., Mezzanzanica, M., Potertì, D., Serino, A., & Seveso, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-662-72243-5_17
CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs
Xu, Y., Ji, S., Zhu, Q., & Che, W. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.45
Financial Question-answering Dataset for Slovak Language Model Evaluation
Hládek, D., Sopkovič, K., Staš, J., Sokolová, Z., & Pleva, M. (2025) · Journal of Linguistics/Jazykovedný casopis · doi:10.2478/jazcas-2025-0022
Structured Pruning Method for Large Language Models with Adaptive Compression Ratios
Shvets, V., & Shapoval, N. (2025) · Artificial Intelligence · doi:10.15407/jai2025.04.088
Breaking Myths in LLM scaling and emergent abilities with a comprehensive statistical analysis
Sun, K., & Wang, R. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132542 · Cited by 13
Z -Prunear: Post-Training Pruning of Large Language Models for Efficiency without Retraining
Bhuiyan, S. B., Adib, M. S. H., Bhuiyan, M. A., Kabir, M. R., Farazi, M. R., Rahman, S., & Mohammed, N. (2025) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa66935.2025.11315381 · Cited by 3
“To assess zero-shot performance, we evaluated on several benchmarks, including HellaSwag [38], BoolQ [5], WinoGrande [35], MNLI [38], and WNLI [26].”
SallyEval: A Scalable Evaluation System for Ranking LLMs with Qualitative Feedback
Kouros, C. E., Tegos, S. D., Tassios, A., Manousaridis, K., Bouas, C., Παπουτσόγλου, Μ., Καλτσά, Μ., Mavropoulos, T., Vrochidis, S., & Μεδίτσκος, Γ. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-14138-5_3
Information et intelligence artificielle
Nesvijevskaia, A., & Berechet, S. (2025) · De Boeck Supérieur eBooks · doi:10.3917/dbu.broud.2025.01.0147
Greek Automatic Text Summarization: From General Language Models to Legal Domain Challenges
Αθανασίου, Π., Karvelis, P. S., & Stylios, C. D. (2025) · South-East Europe Design Automation, Computer Engineering, Computer Networks and Social Media Conference (SEEDA-CECNSM) · doi:10.1109/seeda-cecnsm68644.2025.11329930
“…question answering accuracy from passages[28]. c) HellaSwag – measures commonsense reasoning and completion of challenging narrative scenarios [29]. d) ARC-Challenge – tests scientific and general knowledge reasoning in multiple-choice format [30]. e) TruthfulQA – evaluates truthfulness and…”
H-RWKV: Hybrid model architecture for RWKV
Yuan, X., Li, H., Yang, J., Liu, X., & Gao, T. (2025) · International Conference on Artificial Intelligence and Foundation Model · doi:10.1145/3786709.3786721
“We evaluated the models on several nat-ural language task sets, including PIQA, MMLU, SIQA, ARC-E[3], HELLASWAG[19], OPENBOOKQA[12], and WINOGRANDE[15].”
Kurz, S., Chen, J.-J., Flek, L., & Zhao, Z. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.599 · Cited by 7
“Additionally, we evaluate on the multilingual mARC, mMMLU and mHellaSwag (Lai et al., 2023) datasets as well as their original versions, ARC (Clark et al., 2018), MMLU (Hendrycks et al., 2021) and Hel-laSwag (Zellers et al., 2019) in English, due to their lower sensitivity to the choice of calibration samples (William…”
Hierarchical Subspace Adaptation for Efficient Fine-Tuning of Large Language Models
Dong, J., Wang, Y., Wang, S., & Chen, X. (2025) · International Conference on Machine Intelligence and Nature-Inspired Computing (MIND) · doi:10.1109/mind67540.2025.11351573
“We employed eight datasets covering different types of questions: BoolQ [34], PIQA [35], SIQA [36], HellaSwag [37], WinoGrande [38], ARC-e & ARC-c [39], and OBQA [40].”
RoMeo: Mitigating Dual-dimensional Outliers with Rotated Mixed Precision Quantization
Zhang, Q., Tang, M., Zhai, M., Lei, K., & Zhai, J. (2026) · ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming · doi:10.1145/3774934.3786419
“We conduct both perplexity evaluation on WikiText2 [28] and zero-shot evaluation on six common downstream tasks, including ARC (Challenge and Easy) [8], HellaSwag (HS) [46], LAMBADA [29], PIQA [4], and Wino-Grande (WG) [31].”
Transparent and Equitable Metrics and Models
Arias-Duart, A., Cortés, Á., & Ulises Cortés, A. (2026) · Springer eBooks · doi:10.1007/978-3-031-61050-9_28-1
Zhao, W. X., Zhou, K., Li, J., Tang, T., & Wen, J.-R. (2025) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-981-96-6259-3_12
Conformal Prediction in the Age of Multimodal Foundation Models: A Survey
Stavan, C., Tilaganji, K., Krishnan, S. M., Kancheti, S. S., & Balasubramanian, V. N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15120-9_13
Sutra-PCM: Auditable Commonsense Reasoning with Architecturally Integrated Nyāya Explanations
Basu, A., & Chakraborty, P. (2025) · International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN) · doi:10.1109/icrcicn68210.2025.11364886
“We use a mix of datasets for training, covering three types of reasoning: • Commonsense/Social: SWAG [1], HellaSwag [20], So-cialIQA [2], etc. • Ethical/Moral: ETHICS dataset [21].”
The Benchmark Illusion: How Bounded AI Performance Becomes Unsupported Institutional ProofPreprint
Devin, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6135667
Geng, X., Liu, S., Wang, H., Liu, L., Han, J., & Jiang, H. (2026) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2026.3663251
“L ARGE language models (LLMs) have extensively been applied to a variety of natural language processing tasks [1], [2], [3], [4], and achieved remarkable success.”
MDLA: Multi-Head Latent Linear Attention for Low-Complexity and Memory-Efficient KV Cache
Kim, N., Lee, J., & Lee, S. (2026) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.1109/iceic69189.2026.11386168
“The benchmarks included MMLU (5-shot) [23], HellaSwag [24], ARC (Easy and Challenge) [25], and TruthfulQA [26], covering a broad range of reasoning, commonsense, and factual understanding tasks.”
MOEVIL: Poisoning Experts to Compromise the Safety of Mixture-of-Experts LLMs
Kim, J., Na, S. H., Song, M., Shin, S., & Son, S. (2025) · Annual Computer Security Applications Conference · doi:10.1109/acsac67867.2025.00091
“For HellaSwag [69] (Reason), we use the evaluation prompts of instruction-tuned Llama 3.2 models [36] and measure the accuracy of generated answers for sentence completion tasks by selecting the best completion among A , B , C , and D .”
InfluentialNanda, S., Majee, A., & Iyer, R. (2025) · Proceedings · doi:10.1109/icdm65498.2025.00154
“SST5 [61] involves 5-way classification, DBPedia [62] features 14-way classification, and HellaSwag [63] requires 4-way classification.”
Khuwaileh, B. A., & Matesha, P. (2026) · Annals of Nuclear Energy · doi:10.1016/j.anucene.2026.112223 · Cited by 1
Guan, Y., Cui, Y., Zhao, L., Shen, G., Chen, Y., & Guo, C. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00541-9 · Cited by 1
“For the zero-shot evaluation, we utilize different benchmark datasets, including PIQA (Bisk et al 2020), ARC (Clark et al 2018), HellaSwag (Zellers et al 2019) and WinoGrande (Sak-aguchi et al 2020).”
InfluentialDeveloping a Testing Framework for Applications of LLMs within Islamic Finance
Al-Syed, Q., & Mahomed, Z. (2025) · World Scientific Annual Review of Islamic Finance · doi:10.1142/s2811023425500029
Rodt, V., & Domahs, F. (2026) · Yearbook of the German Cognitive Linguistics Association · doi:10.1515/gcla-2025-0101
“Despite notable advancements demonstrated by LLMs, such as GPT-4 (OpenAI 2023), several well-established language understanding benchmarks (e.g., Clark et al. 2018; Zellers et al. 2019; Hendrycks et al. 2020) do not include dedicated tests for assessing the comprehension of sarcasm or irony.”
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode Inference
Tang, X., Meng, F., Tang, P., Wang, Y., Yin, D., Sun, X., & Zhang, M. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790237 · Cited by 5
“Performance is measured with the LightEval framework on MMLU [55], ARC (Easy/Challenge) [56], PIQA [57], HellaSwag [58], OpenBookQA (OBQA) [59], and WinoGrande (WG) [60].”
SVDLoRA: Data-Driven Low-Rank Adaptation via Spectral Decomposition
Zhang, F., Shen, S., Bi, C., Su, L., Huang, Q., & Wang, S. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00175
“• HellaSwag [23]: Evaluates a model’s ability to select the most plausible continuation for a given context.”
Multi-Stage Variance-Controlled Gradient Updates: Toward Robust Continual Learning
Feng, J., Zhang, Q., Xu, Y., Li, P., & Yang, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7072-0_9
WLR: Well-conditioned linear reconstruction for retraining-free pruning of LLMs
Li, S., Xiang, J., Wei, J., Zhu, C., Yang, J., Chen, J., Yang, J., Wei, X., Jiang, Y., & Liu, Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108840
CLAIM: Mitigating Catastrophic Forgetting in Continual Instruction Fine-Tuning Large Language Models
Qiu, X., Shi, S., Hao, T., & Tan, X. (2026) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2026.3674105
“Furthermore, to evaluate the potential decline in domain knowledge and reasoning capabilities, we employ the massive multitask language understanding benchmark (MMLU) [63], [64] and Hellaswag [65], both of which are well-regarded LLMs evaluation benchmarks.”
A Benchmark Dataset for Bus Travel and Dwell Time Prediction
Horn, A., Adam, P.-R., & Schmidtner, S. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423733
“Because comparability is important in machine learning research, in the fields of computer vision and natural language processing, standardized benchmarks are used (e.g., Ima-geNet [3], HellaSwag [4]).”
Zhang, K., Zhang, Y., Zheyang, L. I., Tan, L., Tan, W., Re, Y., Hu, J., & Xu, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6434892
SWSC: Compressing Large Language Models via Sharing Weights for Similar Channels
Zeng, B., Tang, Y., Ji, B., Liu, X., Li, X., & Yu, J. (2025) · doi:10.1145/3788149.3788206
“…evaluate the performance of the compressed model on complex tasks such as common-sense reasoning and language understanding, we selected PIQA [2], ARC [4] (including ARC-E and ARC-C), BOOLQ [3], HellaSwag [25], and WinoGrande [18] from the natural language processing (NLP) field as benchmark tests.”
BATQuant: Outlier-Resilient MXFP4 Quantization via Learnable Block-Wise Optimization
Li, J.-F., Zhang, M., Xia, X., Bao, H., Bai, H., Dong, Z., & Yu, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37010-5_37 · Cited by 6
Analyzing AI Evaluation Benchmarks Through Information Retrieval and Network Science
Simeoni, G., Soprano, M., Lunardi, R., Roitero, K., & Mizzaro, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21300-6_25
Misspellings in natural language processing: A survey of recent literature
Sperduti, G., & Moreo, A. (2026) · Natural language processing. · doi:10.1017/nlp.2026.10020 · Cited by 8
One-Shot Training for Reduction of Performance Variance in LLM Rotation Quantization
Park, S., Kim, C., & Chung, K.-S. (2026) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic68212.2026.11454275
“For evaluation, we measured perplexity on WikiText-2 [7] and employed eight zero-shot commonsense reasoning tasks: PIQA [8], HellaSwag [9], WinoGrande [10], ARC-easy and ARC-challenge [11], OpenBookQA [12], BoolQ [13], and SIQA [14].”
Multi-dimensional Prompt Expansion: Black-Box DoS Attacks on Multi-agent LLMs
Wang, X., Zhang, T., & Zhu, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-8420-8_22
OOQ: Outlier-Oriented Quantization for Efficient Large Language Models
Wang, H., Liu, B., Shao, H., Xiao, B., Zeng, K., Wan, G., & Qian, Y. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434611
“To evaluate downstream task capabilities, we tested on a diverse range of zero-shot benchmarks, including WinoGrande [32] , PiQA [33] , HellaSwag [34], ARC (comprising ARC-easy and ARC-challenge) [35] and BoolQ [36].”
Poimanov, D. R., & Shutov, M. S. (2025) · Automatic Documentation and Mathematical Linguistics · doi:10.3103/s0005105525701389
Analysis and control of online interactions through neural natural language processing
Laugier, L. (2022) · theses.fr (ABES) · doi:10.70675/4743aa15z209az4addz8f29z95150aa230b8
Efficient Neural Networks : Post Training Pruning and Quantization
Yvinec, E. (2023) · theses.fr (ABES) · doi:10.70675/9f61bfc9zfda1z4803z99a4z3834d6e2212a · Cited by 1
Advancing fairness in natural language processing : from traditional methods to explainability
Jourdan, F. (2024) · theses.fr (ABES) · doi:10.70675/b8535a05z82f1z46f7z9688z3cd35cea8381 · Cited by 3
The Inefficient Pricing of News ShocksPreprint
Pourmohammadi, M., Didisheim, A., Kelly, B. T., & Tian, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6540399
Vella, S., Sharieh, S., Hussain, F., & Ferworn, A. (2025) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet63943.2025.11472552
“HellaSWAG [20] is a more challenging version which uses grounded truth from WikiHow and ActivityNet instead of LLM-generated incorrect answers.”
Modi, S., Vishwakarma, R., Rajeev, R., Parikh, T., Patel, K., & Diamond, H. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476522
“We evaluate on standard LLM benchmarks: BBH [20] (3-shot), DROP [21] (3-shot), GSM8K [22] (8-shot), HellaSwag [23] (10-shot), MATH [24] (4-shot), and MMLU [25] (5-shot).”
LMSQuant: Learnable Multiscale Post-training Quantization for Large Language Models
Yang, Y., Zhang, Y., & Zhang, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7078-2_11
MAR: Efficient Large Language Models Via Module-Aware Architecture Refinement
Cai, J., Wang, G., Zhao, K., Tang, J., Wang, X., Leng, L., Cheng, R., Ma, Y., & Guo, Q. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464338
“We conduct zero-shot evaluations on six commonsense reasoning and question answering datasets: Physical Interaction QA (PIQA) [26], Boolean Questions (BoolQ) [27], Winogrande (WG) [28], HellaSwag (HS) [29], ARC Challenge (ARC-C), and ARC Easy (ARC-E)[30].”
Collaborative Compression for Large-Scale MoE Deployment on Edge
Chen, Y., Xie, Y., Yang, R., Jiang, W., Wang, W., He, Y., Chen, Y., Zhao, P., & Wang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463800 · Cited by 3
Zhu, C., Jiang, H., Liu, J., Zhao, K., & Zhang, R. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463074
S 2 VD: A Subspace-Aware SVD Method for Efficient LLM Compression
Zhou, H., Guan, J., Zhang, J., Wei, R., Zhao, X., Jiang, Z., & Ji, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463557
On the Importance of a Multi-Scale Calibration for Quantization
Son, S., Seong, I., Kim, J., Jang, H., & Jeon, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463220
Two-Stage Grid Optimization for Group-Wise Quantization of LLMs
Kim, J., Lee, G., Son, S., Kim, J., & Jeon, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464464
Alternating Balancing Sums for Accurate Low-Power Dot Products
Natesh, V., & Kung, H. T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464192
“We compare ABS to prior methods when running inference computations from the MLP layer of several 8-bit quantized LLMs on GSM8K [23], Hellaswag [24], MMLU [25], and ANLI [26] datasets.”
DIRA: Deep High-Rank Adaptation of Pre-Trained Language Models
Cao, S., Zhang, D., Zhang, X., Wang, J., & Zhou, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463927
Parameter Localization and Relearning for Safety Disalignment in Large Language Models
Zhou, Y. J., Xing, W., Lin, C., Kong, D., Hu, C., & Han, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463981
Liu, X. R., Wang, S., Li, Y., & Li, K. (2026) · Intelligent Data Analysis · doi:10.1177/1088467x261441866
Sparsity Induction for Accurate Post-Training Pruning of Large Language Models
Jiang, M., Li, Z., Liu, X., Zhang, J., Chen, M., & Gu, Q. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461222
CORM: Coarse-To-Fine-Grained Offloading for Smoe LLM Inference on Consumer-Grade GPU
Tao, Z., Peng, J., Zhu, H., Zhang, W., Chen, T., & Zhang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461364
Preserving Knowledge in Large Language Models Via Model-Agnostic Intrinsic Generative Replay
Zhang, Z., Sun, Y., Zhao, T., Sha, L., Xu, R., Lee, K., & Yin, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464789
Liu, T. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6586558
Zhou, Z., Chen, C., Jin, Z., Qu, B., Jiang, H., Tian, X., & Lin, X. (2026) · International Journal of General Systems · doi:10.1080/03081079.2026.2663918
Low Bit-Width LLM Acceleration via Symmetric Lookup Format and Compute-in-Decoding Paradigm
Zou, Z., Chen, X., Lin, J., Hu, B., Yan, X., Chen, S., Zhang, C., Kang, X., Cai, H., & Liu, B. (2026) · IEEE Transactions on Computers · doi:10.1109/tc.2026.3689022
Layer-wise heterogeneity-guided heterogeneous pruning: An LLM compression method
Yu, Z., Ma, Z., Shen, G., Chen, Y., Guo, C., Duan, X., Qian, Q., & Cui, Y. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00724-4
KEN: An Execution Engine for Unstructured Database Systems
Kossmann, F., Wu, Z., Turk, A., Tatbul, N., Cao, L., & Madden, S. R. (2026) · Proceedings of the VLDB Endowment · doi:10.14778/3796195.3796204
“HellaSwag [93] is a popular common-sense reasoning benchmark.”
Ma, J., & Tao, H. (2026) · International Conference on Generative Artificial Intelligence and Digital Media · doi:10.1145/3803730.3803762
Structured Pruning of LLMs for Resource-Constrained Devices: A Survey
Ahmed, D., & Abbas, G. (2026) · Global Conference on Wireless and Optical Technologies (GCWOT) · doi:10.1109/gcwot69191.2026.11499662
“• Knowledge/reasoning benchmarks such as MMLU [28], HellaSwag [29], TruthfulQA [30], and GSM8K [31] stress factual knowledge and reasoning; they are sensitive to global, long-range, or arithmetic capabilities that pruning can disproportionately damage and therefore are necessary when assessing…”
Brom, P. C., Oliveira, V. D., & Weigang, L. (2026) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2026.6343 · Cited by 1
“We sample k = 15 base items per benchmark (ARC, HellaSwag, MMLU).”
InfluentialLinyi, Z., Yuping, P., Xun, G., & Yong, Z. (2026) · Asia Conference on Algorithms, Computing and Machine Learning (CACML) · doi:10.1109/cacml68972.2026.11507008
“We evaluate at 2400 steps on seven benchmarks us-ing lm-evaluation-harness [34]: HellaSwag [35] (0-shot), ARC-Easy/Challenge [36] (0/5-shot), WinoGrande [37] (5-shot), PIQA [38] (0-shot), BoolQ [39] (0-shot), and MMLU [40] (5-shot).”
Liu, D., Yu, Y., & Wu, Y. N. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.10.724161
EvasionEval: A Benchmark for LLMs in Evaluating Advanced Defense Evasion Techniques
Wang, B., Huang, Y., Li, X., Yu, J., & Zhang, X. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-23450-6_20
Can LLMs Guess Like Humans? Evaluating Inferential Reasoning Under Ambiguity
Radhika, G., Ravikiran, M., Sharma, T., & Saluja, R. (2026) · Studies in Rhythm Engineering · doi:10.1007/978-3-032-19966-9_7
Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators
Sridhar, A., & Johansen, A. R. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813146
Cross-sectional analysis of large language models in current natural language generation challenges
Maestre, M. M., Martínez-Murillo, I., Martínez, A. M., & Lloret, E. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104922
When bots mislead markets: asymmetric contamination risk in sentiment-based ınvestment decisions
Teti̇k, M. (2026) · Future Business Journal · doi:10.1186/s43093-026-00863-0
Donvito, A., & Lieto, A. (2026) · Cognitive Systems Research · doi:10.1016/j.cogsys.2026.101473 · Cited by 1
“HellaSwag, [Zellers et al. (2019)]: choosing the best ending to a story or set of instructions, with adversarially mined examples in order to make the task more complicated for the model (79.3% accuracy); 3.”
InfluentialWhen Do Large Language Models (LLMs) Struggle to Count Letters?
Fu, T., Ferrando, R., Conde, J., Arriaga, C., & Reviriego, P. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3818606 · Cited by 3
CALSI: Context-Aware Layer Skipping Inference for On-Device LLM Serving
Luo, P., Yuan, M., Zhang, N., Zhong, R., Zhang, J., Hu, C., & Li, X. (2026) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2026.3697502
“Besides, we test the common sense reasoning ability of the LLM (e.g. multiple choice questions) [33], including PIQA [34], ARC-Easy [35], ARC-Challenge [35], HellaSwag [36], BoolQ [37], Winogrande [38] and COPA [39].”
Pagliaro, A. (2026) · Electronics · doi:10.3390/electronics15112428
Dai, C., Ye, C., Li, Z., Tao, J., Zhang, W., Zhang, G. L., & Li, X. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539743
“[23], ARC-Easy (i.e., “ARC -”
LLM Compression with Jointly Optimizing Architectural and Quantization Choices
La, H.-L., Le, T.-T., Taherkordi, A., & Ha, P. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-36868-3_27
RheoSparse: Exploring Finer Grained Structured Sparsity for Small Language Models
Zheng, J., & Chen, G. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539084
“We then report zero-shot accuracy on seven widely used downstream benchmarks: ARC (Easy/Hard) [14], BoolQ [15], HellaSwag [16], Open-BookQA [17], PIQA [18], WinoGrande [19], and MMLU [20].”
Geng, X., Liu, S., Wang, H., Han, J., & Jiang, H. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539270
“5-0.5B (all integrated with AWQ) on five tasks, LAMBADA [28], PIQA [29], ARC-Challenge [30], ARC-Easy [30], and HellaSwag [31].”
FlattenGPT: Depth Compression for Transformer with Layer FlatteningPreprint
Xu, R., Guo, Q., Zhu, Y., Ji, X., Yang, M., & Zhang, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6890079
Hint-oriented self-suggestive learning for commonsense information processing
Fan, Y., Zou, B., Li, Y., & Hong, Y. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104964
Chen, Z., Wang, W., Zhu, J., Wang, X., Liu, Y., Liu, Z. T., Luo, G., Sun, P., Su, W., Tian, H., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3702168
PerfMamba: Performance Analysis and Pruning of Selective State Space Models
Asif, A. A., Kashaniyan, M., Yu, S., Muñoz, J. P., & Jannesari, A. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-9694-2_3
Bidirectional Alignment Fine-Tuning for Decoderonly Language Models
Vinogradov, A., Li, M., Mkrtchian, G., & Pavlikov, A. (2026) · Wave Electronics and its Application in Information and Telecommunication Systems (WECONF) · doi:10.1109/weconf69969.2026.11553811
Metadata
- Authors
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4791-4800
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2946609015
- Cited by
- 830
- References
- 20
- Citations by year
- 2026: 200 · 2025: 263 · 2024: 109 · 2023: 93 · 2022: 50 · 2021: 61 · 2020: 44 · 2019: 10
- FWCI
- 15.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1703166, 1256082, 1637479, 1524371)
- Nvidia
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- commonsense reasoning, natural language inference, AI-generated text, BERT, benchmark design, pre-trained language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
