The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
HellaSwag: Can a Machine Really Finish Your Sentence?
Open accessAbstract
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys."With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached.Does this mean that machines can perform human level commonsense inference?In this paper, we show that commonsense inference still proves di cult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset.Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle ( †48%).We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers.AF proves to be surprisingly robust.The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.Our construction of HellaSwag, and its resulting di culty, sheds light on the inner workings of deep pretrained models.More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is no 'ground-truth' implementation from the original benchmark authors, and many popular benchmarks (Paperno et al., 2016; Clark et al., 2018; Zellers et al., 2019) that have been 'retrofitted' for compatibility with autoregressive LMs.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zellers et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 1 biasing the comparison to human performance.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Traditional language model benchmarks, such as HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), MMLU (Hendrycks et al., 2021), GSM-8K (Cobbe et al., 2021), and BIG-Bench (Srivastava et al., 2023), primarily assess LLMs on tasks like commonsense reasoning and standardized exams.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“for evaluating the following tasks that measure common sense reasoning: LAMBADA (OpenAI version in LM Evaluation Harness) (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge, ARC-easy (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2021).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Zellers et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Self-citationDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Research works printed but not linked: 2
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for e cient text classification. In Proceedings of the 15th Confer- ence of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 427-431.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Cited by
Sorted by citations
Items 301–400 of 907
Locating and Mitigating Gender Bias in Large Language Models
Cai, Y., Cao, D., Guo, R., Wen, Y., Liu, G., & Chen, E. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5672-8_40 · Cited by 19
Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report
Ostermann, S., Zhang, S., Roth, M., & Clark, P. E. (2019) · doi:10.18653/v1/d19-6007 · Cited by 7
Developing a GPT-3-Based Automated Victim for Advance Fee Fraud Disruption
Hewett, J., & Leeke, M. (2022) · IEEE Pacific Rim International Symposium on Dependable Computing (PRDC) · doi:10.1109/prdc55274.2022.00034 · Cited by 9
The Web Can Be Your Oyster for Improving Language Models
Li, J., Tang, T., Zhao, W. X., Wang, J., Nie, J., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.46 · Cited by 11
Eliciting and Understanding Cross-task Skills with Task-level Mixture-of-Experts
Ye, Q., Zha, J., & Ren, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.189 · Cited by 19
COINS: Dynamically Generating COntextualized Inference Rules for Narrative Story Completion
Paul, D., & Frank, A. (2021) · doi:10.18653/v1/2021.acl-long.395 · Cited by 12
Predict then Interpolate: A Simple Algorithm to Learn Stable ClassifiersPreprint
Bao, Y., Chang, S., & Barzilay, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12628 · Cited by 10
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
Phang, J., Chen, A., Huang, W., & Bowman, S. R. (2022) · doi:10.18653/v1/2022.dadc-1.8 · Cited by 9
A Survey of Post-Training Scaling in Large Language Models
Lai, H., Liu, X., Gao, J., Cheng, J., Qi, Z., Xu, Y., Yao, S., Zhang, D., Du, J., Hou, Z., et al. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2025.acl-long.140 · Cited by 8
CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models
Niu, C., Li, C., Ng, V., & Luo, B. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00055 · Cited by 11
BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
Jiang, Y., Ilievski, F., Ma, K., & Sourati, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.885 · Cited by 29
Frankford, E., Höhn, I., Sauerwein, C., & Breu, R. (2024) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10662990 · Cited by 13
“Additionally, a viable approach to establish a quantitative benchmark for programming exercise generation using LLMs could be a match-based multiple choice approach, similar to the hellaSWAG benchmark [25].”
BackgroundSoliman, G., Zaki, H., & Kilany, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100131 · Cited by 13
Precise length control for large language models
Butcher, B., O’Keefe, M., & Titchener, J. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100143 · Cited by 17
Aligning large language models with human preferences using historical text edits
Majkutewicz, J., & Szymański, J. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113566 · Cited by 9
Harnessing language models for computational literature review of emerging AI topics
Chung, J., Jeong, B., Park, Y.-J., Jeong, H., Lee, J., Yoon, J., & Choi, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104245 · Cited by 7
SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews
Huotala, A., Kuutila, M., & Mäntylä, M. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00053 · Cited by 12
“…& reasoning ARC-Challenge [13] Knowledge & reasoning BIG-Bench-Hard [14] Knowledge & reasoning GPQA [15], Diamond Knowledge & reasoning HellaSwag [16] Knowledge & reasoning MMLU [11], [17] Knowledge & reasoning GSM8K [11], [18] Mathematical & logical reasoning MATH [11], [19] Mathematical &…”
Analyzing Multi-Task Learning for Abstractive Text Summarization
Kirstein, F., Wahle, J. P., Ruas, T., & Gipp, B. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.5 · Cited by 9
Pre‐trained language models: What do they know?
Guimar�ães, N., Campos, R., & Jorge, A. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1518 · Cited by 10
Tunable Soft Prompts are Messengers in Federated Learning
Dong, C., Xie, Y., Ding, B., Shen, Y., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.976 · Cited by 18
RepairBench: Leaderboard of Frontier Models for Program Repair
Silva, A., & Monperrus, M. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00006 · Cited by 23
“HellaSwag [Zellers et al., 2019] focuses on testing models’ commonsense reasoning, while benchmarks like GSM8K [Cobbe et al., 2021] and MATH [Hendrycks et al., 2021] are designed to evaluate models’ mathematical problem-solving capabilities.”
Advanced natural-based interaction for the ITAlian language: LLaMAntino-3-ANITA
Polignano, M., Basile, P., & Semeraro, G. M. (2026) · Scientific Reports · doi:10.1038/s41598-025-31319-0 · Cited by 40
“HellaSwag [32] is another benchmark that focuses on commonsense reasoning, challenging LLMs to complete passages in a way that requires understanding intricate details.”
BackgroundHow Fast can BERT Learn Simple Natural Language Inference?
Lin, Y., & Su, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.51 · Cited by 7
Chapter 13. Generalizable Neuro-Symbolic Systems for Commonsense Question Answering
Oltramari, A., Francis, J., Ilievski, F., Ma, K., & Mirzaee, R. (2021) · Frontiers in artificial intelligence and applications · doi:10.3233/faia210360 · Cited by 8
ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models
Li, J., Tang, T., Gong, Z., Yang, L., Yu, Z., Chen, Z., Wang, J., Zhao, X. W., & Wen, J.-R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.258 · Cited by 8
Kamalloo, E., Rezagholizadeh, M., & Ghodsi, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.84 · Cited by 10
Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks
Wang, Z., Pan, X., Yu, D., Yu, D., Chen, J., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.246 · Cited by 13
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning
Ye, Q., Beltagy, I., Peters, M. E., Ren, X., & Hajishirzi, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.454 · Cited by 8
Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis
Yang, S., Kim, J., Jang, J., Ye, S., Lee, H., & Seo, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00666 · Cited by 12
Aghahadi, Z., & Talebpour, A. (2022) · Journal of Applied Non-Classical Logics · doi:10.1080/11663081.2022.2041352 · Cited by 9
A Method for Building a Commonsense Inference Dataset based on Basic Events
Omura, K., Kawahara, D., & Kurohashi, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.192 · Cited by 8
Chronologically Consistent Large Language ModelsPreprint
He, S., Lv, L., Manela, A., & Wu, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5159615 · Cited by 62
Improving LLM Outputs Against Jailbreak Attacks With Expert Model Integration
Tsmindashvili, T., Kolkhidashvili, A., Kurtskhalia, D., Maghlakelidze, N., Mekvabishvili, E., Dentoshvili, G., Shamilov, O., Gachechiladze, Z., Saporta, S., & Choladze, D. D. (2025) · IEEE Access · doi:10.1109/access.2025.3592458 · Cited by 7
“One interesting point here is that confidence is also a helpful metric: the system can be designed to ignore the expert model’s output when the associated confidence score is low.”
LLM.265: Video Codecs are Secretly Tensor Codecs
Xu, C., Wu, Y., Yang, X., Chen, B., Lentz, M., Zhuo, D., & Wills, L. W. (2025) · IEEE Micro · doi:10.1109/mm.2026.3701735 · Cited by 9
A survey on large language models unlearning: taxonomy, evaluations, and future directions
Le-Khac, U. N., & Truong, V. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11376-7 · Cited by 11
“Various other standard academic and general knowledge benchmarks were also commonly employed, including the multi-question MT-Bench benchmark (Zheng et al. 2023), TruthfulQA (Lin et al. 2021), PubMedQA (Jin et al. 2019), MathQA (Amini et al. 2019), HellaSwag (Zellers et al. 2019), ARC (Yadav et al. 2019), OpenBookQA (…”
MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
Chitty-Venkata, K. T., Howland, S., Azar, G. A., Soboleva, D., Vassilieva, N., Raskar, S., Emani, M. K., & Vishwanath, V. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767706 · Cited by 8
“We benchmark LLMs on nine widely adopted language understanding tasks from the lm-eval [16] [7], HellaSwag [53], MMLU [19], OpenBookQA [33], RTE [45], WinoGrande [37].”
General scales unlock AI evaluation with explanatory and predictive power
Zhou, L., Pacchiardi, L., Martínez‐Plumed, F., Collins, K. M., Moros-Daval, Y., Zhang, S., Zhao, Q., Huang, Y., Sun, L., Prunty, J. E., et al. (2026) · Nature · doi:10.1038/s41586-026-10303-2 · Cited by 21
“Last, we prompted GPT-4o to annotate three quality indicators: (1) the accuracy of ground-truth labels; (2) the objectivity; and (3) the unambiguity, for all instances, graded with a Likert scale from 1 to 5 (Supplementary Tables 24, 25 and 26).”
Implicit knowledge-augmented prompting for commonsense explanation generation
Ge, Y., Yu, H.-T., Lei, C., Liu, X., Jatowt, A., Kim, K., Lynden, S. J., & Matono, A. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-024-02326-w · Cited by 6
“For instance, HellaSwag [33] presents questions where the answer cannot be determined solely from the provided context, forcing models to go beyond surface-level understanding; SocialIQA [8] evaluates a model’s understanding of interpersonal dynamics and social context.”
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
Hu, R., Cheng, Y., Meng, L., Xia, J., Zong, Y., Shi, X., & Lin, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715265 · Cited by 14
“Static benchmarks have been developed to assess the performance of LLMs across various tasks, such as language understanding [12], world knowledge [15, 53], reasoning [49], coding [7], and math [13].”
LoCo: Low-Bit Communication Adaptor for Large-Scale Model Training
Xie, X., Lin, Z., Toh, K.-C., & Zhou, P. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3544764 · Cited by 13
D’Oosterlinck, K., Xu, W., Develder, C., Demeester, T., Singh, A., Potts, C., Kiela, D., & Mehri, S. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00748 · Cited by 31
Mosaic: Composite projection pruning for resource-efficient LLMs
Eccles, B. J., Wong, L., & Varghese, B. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.108056 · Cited by 7
“Accuracy BoolQ [33] Accuracy HellaSwag [34] Norm.”
COSY: COunterfactual SYntax for Cross-Lingual Understanding
Yu, S., Zhang, H., Niu, Y., Sun, Q., & Jiang, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.48 · Cited by 6
On the Generalization Abilities of Fine-Tuned Commonsense Language Representation Models
Shen, K., & Kejriwal, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91100-3_1 · Cited by 6
ONNXPruner: ONNX-Based General Model Pruning Adapter
Ren, D., Li, W., Ding, T., Wang, L., Fan, Q., Huo, J., Pan, H., & Gao, Y. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3554560 · Cited by 10
“We followed the LLaMA evaluation protocol [60] and performed classification on seven diverse datasets: BoolQ [61], PIQA [62], HellaSwag [63], WinoGrande [64], ARC-easy [65], ARC-challenge [65], and OpenbookQA [66].”
He, J., Peng, B., Liao, Y., Liu, Q., & Xiong, D. (2021) · doi:10.18653/v1/2021.acl-long.469 · Cited by 7
“We suggest that this generation task could be reformulated as a multi-choice question answering task by providing alternative rationales as distractors, similar to VCR (Zellers et al., 2019a).”
Naturalistic Causal Probing for Morpho-Syntax
Amini, A., Pimentel, T., Meister, C., & Cotterell, R. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00554 · Cited by 14
Xing, L., Xie, Y., Hu, Y., & Peng, W. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.42 · Cited by 6
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
“There is no “ground-truth” implementation from the original benchmark authors, and many popular benchmarks (Paperno et al., 2016; Clark et al., 2018; Zellers et al., 2019) that have been “retrofitted” for compatibility with autoregressive LMs.”
MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
Cai, W., Qin, L., & Huang, J. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716006 · Cited by 10
“E” indicates the fine-tuned model without fine-tuning all expert parameters, “FT-Full” represents the fine-tuned model with full state saving at each checkpointing, and “FT-PEC” denotes the fine-tuned model utilizing PEC that saves 1/8 of the experts at each checkpoint.”
Hallgarten, P., Hallitschke, V. J., Kasneci, E., Beigl, M., & Große-Puppendahl, T. (2025) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3749552 · Cited by 5
AxCore: A Quantization-Aware Approximate GEMM Unit for LLM Inference
Zou, J., Chen, Y., Chen, X., Xu, C., & Chen, X. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756094 · Cited by 10
“We also evaluate AxCore on four standard zero-shot benchmark datasets (ARC-e [8], HellaSwag [52], Piqa [4], and Winogrande [2]), using the lm-eval-harness framework [17].”
Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
Kim, W., Lee, Y., Kim, Y., Hwang, J., Oh, S., Jung, J., Huseynov, A., Park, W. G., Park, C. H., Mahajan, D., et al. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756121 · Cited by 5
An Overview of Recent Advances in Natural Language Processing for Information Systems
O’Shaughnessy, D. D. (2026) · Applied Sciences · doi:10.3390/app16021122 · Cited by 9
“…step or trajectory level; The Stanford Question Answering Dataset [130] and DROP [131]—often used as reading comprehension benchmarks; HellaSwag [132]—advances common sense NL inference via the use of Adversarial Filtering, which is a data collection paradigm where discriminators iteratively…”
Assessing small language models for code generation: An empirical study with benchmarks
Hasan, M. M., Waseem, M., Kemell, K., Rasku, J., Ala-Rantala, J., & Abrahamsson, P. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112815 · Cited by 23
“Existing studies mostly focus on general natural-language processing tasks [9] [10] [11].”
Ko, E., & Kang, P. (2025) · IEEE Access · doi:10.1109/access.2025.3553870 · Cited by 8
Rethinking Data: Towards Better Performing Domain-Specific Small Language Models
Nazarov, B., Frolova, D., Lubarsky, Y., Gaissinski, A., & Kisilev, P. (2024) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/gcwkshp64532.2024.11101189 · Cited by 5
“Each data unit is represented in standardized format, comprising of five distinct fields: [Question, Options, Answer, Explanation, Category] knowledge and domain-specific knowledge [10], [11], [12].”
Tozadore, D. C., Ertug, N., Chaker, Y., & Abderrahim, M. (2025) · IEEE RO-MAN · doi:10.1109/ro-man63969.2025.11217800 · Cited by 7
“For story completion and reasoning, we considered Hellaswag [16].”
Towards Semantics-Enhanced Pre-Training: Can Lexicon Definitions Help Learning Sentence Meanings?
Ren, X., Sun, X., Wang, H., & Liu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17619 · Cited by 6
Not a Number: Identifying Instance Features for Capability-Oriented Evaluation
Burnell, R., Burden, J., Rutar, D., Voudouris, K., Cheke, L. G., & Hernández‐Orallo, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/392 · Cited by 8
Ramamurthy, P., & Aakur, S. N. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956441 · Cited by 6
The silicon service spectrum: warmth and competence explain people's preferences for AI assistants
Hernandez, I. A., & Chekili, A. (2024) · Frontiers in Social Psychology · doi:10.3389/frsps.2024.1396533 · Cited by 6
Dual-Assessment Driven Pruning: Iterative Optimizing Layer-wise Sparsity for Large Language Model
Sun, Q., Wang, W., Yanni, Z., He, S., Yi, H., Cai, Z., & Liu, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671780 · Cited by 5
“In terms of zero-shot tasks, we employ the Accuracy metric for zero-shot evaluations on seven benchmarks, including BoolQ[37], RTE[38], HellaSwag[39], WinoGrande[40], ARC-Easy and ARC-Challenge[41] and OpenbookQA[42].”
On the uncertainty principle of neural networks
Zhang, J.-J., Zhang, D., Chen, J., Pang, L.-G., & Meng, D. (2025) · iScience · doi:10.1016/j.isci.2025.112197 · Cited by 8
2-D Transformer: Extending Large Language Models to Long-Context With Few Memory
He, X., Liu, J., & Duan, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3548047 · Cited by 5
“For downstream short-context tasks, we leverage TruthfulQA [49] and HellaSwag [50] datasets to evaluate the model’s understanding of short-context scenarios.”
COMET: Towards Practical W4A4KV4 LLMs Serving
Liu, L., Cheng, L., Ren, H., Zhaohui, X., Pan, Y., Wang, M., Li, X., Han, Y., & Wang, Y. (2025) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3676641.3716252 · Cited by 9
“We evaluate the performance of the COMET inference system at two different levels: kernel-level benchmarking and end-to-end inference performance.”
Jurassic is (Almost) All You Need: Few-Shot Meaning-to-Text Generation for Open-Domain Dialogue
Reed, L., Li, C., Ramirez, A. M., Wu, L., & Walker, M. A. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-5538-9_7 · Cited by 5
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
Zhu, W., Wang, X. E., Yan, A., Eckstein, M. P., & Wang, W. Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.6 · Cited by 6
Knowledge Rumination for Pre-trained Language Models
Yao, Y., Wang, P., Mao, S., Tan, C., Huang, F., Chen, H., & Zhang, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.206 · Cited by 6
Empowering Federated Learning for Massive Models with NVIDIA FLARE
Roth, H. R., Xu, Z., Hsieh, Y.-T., Renduchintala, A., Yang, I., Zhang, Z., Wen, Y., Yang, S., Lu, K. N., Kersten, K., et al. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-78841-3_1 · Cited by 13
Jo, S., & Trummer, I. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3725356 · Cited by 4
“In our experimental evaluation, we measure the cost savings of SpareLLM on NLP tasks using five real-world datasets: MMLU [14], IMDB [24], SMS-Spam [4], AgNews [43], and HellaSwag [41].”
InfluentialLiu, B., Cui, Z., Hu, S., Li, X., Lin, H., & Zhang, Z. (2025) · IEEE Joint International Information Technology and Artificial Intelligence Conference (ITAIC) · doi:10.1109/itaic64559.2025.11163350 · Cited by 15
“Cross-domain benchmarks such as MMLU [4] and HellaSwag [5] introduced systematic evaluation of multitask understanding through carefully curated question sets.”
Assis, G., Freitas, C., & Paes, A. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5814 · Cited by 5
“For instance, HellaSwag [Zellers et al ., 2019] is a multiple-choice question-answering dataset designed to measure models’ commonsense natural language inference capabilities, requiring them to select the most coherent sentence completion.”
LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment
Zeng, B., Ji, B., Liu, X., Yu, J., Li, S., Ma, J., Li, X. S., Wang, S., Hong, X., & Tang, Y. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228461 · Cited by 4
Ramachandran, A., Kundu, S., Raha, A., Kundu, S., Mathaikutty, D. K., & Krishna, T. (2025) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1109/islped65674.2025.11261770 · Cited by 4
“Since OWL can explore different values of N for a fixed M, we fix M=8.”
SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
Czakó, P., Kertész, G., & Szénási, S. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342731 · Cited by 4
“Our evaluations cover language generation tasks using perplexity scores (PPL) on Wikitext-2 [23] and C4 [24] datasets, an accuracy on six zero-shot commonsense reasoning tasks including PIQA [25], WinoGrande [26], HellaSwag [27], LAMBADA [28], Arc-Easy and Arc-Challenge [29].”
HADAD: Hexagonal A-Star with Differential Algorithm Designed for weather routing
de la Jara, J. J., Precioso, D., Bu, L., Redondo-Neble, M. V., Milson, R., Ballester‐Ripoll, R., & Gómez‐Ullate, D. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.120050 · Cited by 4
SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
Liu, H., Gao, H., Zhang, X., Li, C., Zhang, F., Wang, W. W., Ma, F., & Yu, H. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709287 · Cited by 4
“For zero-shot experiments, we measure different models by quantizing the popular LLaMA model [37] from 7 billions to 30 billions on PIQA [3], ARC-easy and ARC-challenge [2], HellaSwag [44] and WinoGrande [34] datasets.”
LLM-Eraser: Optimizing Large Language Model Unlearning through Selective Pruning
Zhang, S., Zhang, L., Zhou, J., Zheng, Z., & Xiong, H. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709312 · Cited by 7
“Specifically, we’re referring to the following: Arithmetic [6], BoolQ [11], Hel-laSwag [55], OpenbookQA(OBQA) [32], and PIQA [4].”
HAAN: A Holistic Approach for Accelerating Normalization Operations in Large Language Models
Peng, T., Xia, T., Qin, J., & Zhang, S. Q. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993005 · Cited by 4
“Next, we present the hardware system evaluation in Section V-B.”
Gignac, G. E., & Ilić, D. (2025) · Intelligence · doi:10.1016/j.intell.2025.101922 · Cited by 4
Jiang, H., & Liu, Y. (2026) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2026.102216 · Cited by 4
Improving the robustness of NLI models with minimax training
Korakakis, M., & Vlachos, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.801 · Cited by 5
CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
Ismayilzada, M., Paul, D., Montariol, S., Geva, M., & Bosselut, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.607 · Cited by 14
EconBERTa: Towards Robust Extraction of Named Entities in Economics
Lasri, K., de Castro, P. V. Q., Schirmer, M., Martin, L. E. S., Wang, L., Dulka, T., Naushan, H., Pougué-Biyong, J., Legovini, A., & Fraiberger, S. P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.774 · Cited by 3
Ye, S., Jang, J., Kim, D., Jo, Y., & Seo, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.822 · Cited by 10
How Well Do LLMs Understand DEECo Ensemble-Based Component Architectures
Töpfer, M., Khalyeyev, D., Bureš, T., Hnětynka, P., & Plášil, F. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-75107-3_13 · Cited by 4
LONG EXPOSURE: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
Wang, T., Li, K., Hao, Z., Bai, D., Ren, J., Zhang, Y., Cao, T., & Yang, M. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41406.2024.00081 · Cited by 6
“…Winogrande [42] Physical interactions understanding RTE [43] Natural language understanding COPA [44] Commonsense causal reasoning HellaSwag [45] Natural language commonsense we conduct our ablation studies and accuracy validation using LoRA owing to its broad acceptance within the domain.”
MotIF: Motion Instruction Fine-Tuning
Hwang, M., Hejna, J., Sadigh, D., & Bisk, Y. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3527290 · Cited by 7
“While VLMs have demonstrated a strong understanding of physical and causal commonsense reasoning [34], [35] and semantic grounding [14], [15], [36], they often work with static images as inputs, and cannot reason about sequential inputs necessary for dynamic tasks.”
Self-citationBuilding UniGPT: A Customizable On-Premise LLM-Solution for Universities
Radas, J., Risse, B., & Vogl, R. (2025) · EPiC series in computing · doi:10.29007/jv1l · Cited by 6
Zesch, T., Hanses, M., Seidel, N., Aggarwal, P., Veiel, D., & de Witt, C. (2024) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet61869.2024.10837635 · Cited by 5
RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation
Liu, J., Kong, Z., Dong, P., Shen, X., Zhao, P., Tang, H., Yuan, G., Niu, W., Zhang, W., Lin, X., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889613 · Cited by 10
“The commonsense reasoning tasks included BoolQ [45], PIQA [46], SIQA [47], HellaSwag [48],WinoGrande [49], ARC-e [50], ARC-c [50], and OBQA [51].”
Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models
Wallace, T., Ombuki-Berman, B. M., & Ezzati‐Jivan, N. (2025) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3676151.3719379 · Cited by 6
Xie, T., Zhao, J., Wan, Z., Zhang, Z., Wang, Y., Wang, R., Huang, R., & Li, M. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132610 · Cited by 4
“6, an acceptable performance threshold can be empirically established to define the critical region where computational errors significantly impact LLM performance.”
It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multiobjective Optimization
Li, B., Di, Z., Yang, Y., Qian, H., Yang, P., Hao, H., Tang, K., & Zhou, A. (2025) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2025.3613937 · Cited by 8
“For the final evaluation of our models, we employed a range of datasets tailored to assess various capabilities, including C-EVAL [64], MMLU [65], GSM8K [66], HellaSwag [67], HumanEval [68], MBPP [69], WinoGrande [70], and BIG-bench hard (BBH) [71].”
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.38 · Cited by 7
A Large Harvested Corpus of Location Metonymy
Mathews, K. A., & Strube, M. (2020) · Language Resources and Evaluation · Cited by 4
Evaluating Humorous Response Generation to Playful Shopping Requests
Shapira, N., Kalinsky, O., Libov, A., Shani, C., & Tolmach, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28238-6_53 · Cited by 5
Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers
Zhong, W., Ma, T., Wang, J., Yin, J., Zhao, T., Lin, C.-Y., & Duan, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.480 · Cited by 4
Exploring the Curious Case of Code Prompts
Zhang, L., Dugan, L. D., Xu, H., & Callison-Burch, C. (2023) · doi:10.18653/v1/2023.nlrse-1.2 · Cited by 6
Metadata
- Authors
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4791-4800
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2946609015
- Cited by
- 830
- References
- 20
- Citations by year
- 2026: 200 · 2025: 263 · 2024: 109 · 2023: 93 · 2022: 50 · 2021: 61 · 2020: 44 · 2019: 10
- FWCI
- 15.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1703166, 1256082, 1637479, 1524371)
- Nvidia
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- commonsense reasoning, natural language inference, AI-generated text, BERT, benchmark design, pre-trained language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
