The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
HellaSwag: Can a Machine Really Finish Your Sentence?
Open accessAbstract
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys."With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached.Does this mean that machines can perform human level commonsense inference?In this paper, we show that commonsense inference still proves di cult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset.Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle ( †48%).We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers.AF proves to be surprisingly robust.The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models.Our construction of HellaSwag, and its resulting di culty, sheds light on the inner workings of deep pretrained models.More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is no 'ground-truth' implementation from the original benchmark authors, and many popular benchmarks (Paperno et al., 2016; Clark et al., 2018; Zellers et al., 2019) that have been 'retrofitted' for compatibility with autoregressive LMs.”
From Lessons from the Trenches on Reproducible Evaluation of Language Models (2024) · cites this paper as Zellers et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, benchmarks are often adversarially selected for tasks that current models struggle with compared to humans, 1 biasing the comparison to human performance.”
From Measuring AI Ability to Complete Long Software Tasks (2025) · cites this paper as [4]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“During RLHF fine-tuning, we observe performance regressions compared to GPT-3 on certain public NLP datasets, notably SQuAD (Rajpurkar et al., 2018), DROP (Dua et al., 2019), HellaSwag (Zellers et al., 2019), and WMT 2015 French to English translation (Bojar et al., 2015).”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“Traditional language model benchmarks, such as HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), MMLU (Hendrycks et al., 2021), GSM-8K (Cobbe et al., 2021), and BIG-Bench (Srivastava et al., 2023), primarily assess LLMs on tasks like commonsense reasoning and standardized exams.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Zellers et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“for evaluating the following tasks that measure common sense reasoning: LAMBADA (OpenAI version in LM Evaluation Harness) (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), ARC-challenge, ARC-easy (Clark et al., 2018), WinoGrande (Sakaguchi et al., 2021).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Zellers et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Williams, S., Waterman, A., & Patterson, D. A. (2009) · Communications of the ACM · doi:10.1145/1498765.1498785 · Cited by 2,798
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Self-citationDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279
Annotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
Research works printed but not linked: 2
- Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for e cient text classification. In Proceedings of the 15th Confer- ence of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 427-431.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Cited by
Sorted by citations
Items 601–700 of 908
Is Multimodal Vision Supervision Beneficial to Language?
Madasu, A., & Lal, V. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00263 · Cited by 1
SharPT: Shared Latent Space Prompt Tuning
Pang, B., Yavuz, S., Xiong, C., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.92 · Cited by 1
TaskWeb: Selecting Better Source Tasks for Multi-task NLP
Kim, J., Asai, A., Ilharco, G., & Hajishirzi, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.680 · Cited by 5
A Data-Driven Study of Commonsense Knowledge using the ConceptNet Knowledge BasePreprint
Shen, K., & Kejriwal, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14084 · Cited by 4
Go Beyond Plain Fine-tuning: Improving Pretrained Models for Social CommonsensePreprint
Chang, T.-Y., Liu, Y., Gopalakrishnan, K., Hedayatnia, B., Zhou, P., & Hakkani-Tür, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05913 · Cited by 1
Boosting coherence of language models.Preprint
Malkin, N., Wang, Z., & Jojić, N. (2021) · arXiv (Cornell University) · Cited by 1
Infinite-parameter Large Language ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-7khtw
Argument Mining on Clinical Trials
Mayer, T. (2020) · theses.fr (ABES) · doi:10.70675/8c2cf7b5z08cbz40e4za421z0115aa4a8eaa · Cited by 1
Evaluating and Improving the Reasoning Abilities of Language Models
Helwe, C. (2024) · theses.fr (ABES) · doi:10.70675/7fc70e59zdd27z4b0fza38dz03a2c67cda50 · Cited by 1
MiMo-V2-Flash Technical Report
Core Team, Xiao, B., Xia, B., Yang, B., Gao, B., Shen, B., Zhang, C., He, C., Lou, C., Luo, F., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.02780 · Cited by 1
“We evaluate MiMo-V2-Flash-Base on a series of benchmarks, encompassing various capabilities: (1) General language understanding and reasoning, including BBH (Suzgun et al., 2023), MMLU (Hendrycks et al., 2021a), MMLU-Redux (Gema et al., 2024), MMLU-Pro (Wang et al., 2024), DROP (Dua et al., 2019), ARC (Clark et al., 2…”
Gated Delta Networks: Improving Mamba2 with Delta Rule
Yang, S., Kautz, J., & Hatamizadeh, A. (2024) · arXiv · doi:10.48550/arxiv.2412.06464 · Cited by 1
“; Zellers et al., 2019), WinoGrande (Wino.”
Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection
Ahuja, K., Sclar, M., & Tsvetkov, Y. (2025) · arXiv · doi:10.48550/arxiv.2504.11900 · Cited by 1
ARB-LLM: Alternating Refined Binarizations for Large Language Models
Li, Z., Yan, X., Zhang, T., Qin, H., Xie, D., Tian, J., shi, Z., Kong, L., Zhang, Y., & Yang, X. (2024) · arXiv · doi:10.48550/arxiv.2410.03129 · Cited by 1
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Hatamizadeh, A., Choi, Y., & Kautz, J. (2026) · arXiv
“Language modeling and common-sense reasoning Table 2 reports WikiText and LAMBADA perplexity [27, 28], zero-shot LAMBADA accuracy, and the common-sense suite from PIQA through BoolQ [29, 30, 31, 32, 33, 34, 35].”
Self-citationBehrouz, A., Razaviyayn, M., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2504.13173 · Cited by 65
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
von Oswald, J., Scherrer, N., Kobayashi, S., Versari, L., Yang, S., Mittal, S., Schlegel, M., Maile, K., Schimpf, Y., Sieberling, O., et al. (2025) · arXiv
“This includes Lambada [74], HellaSwag [114] and RACE-{M,H} [59].”
On the Way to LLM Personalization: Learning to Remember User Conversations
Magister, L. C., Metcalf, K., Zhang, Y., & ter Hoeve, M. (2024) · arXiv · doi:10.48550/arxiv.2411.13405 · Cited by 6
“We also rely on a number of benchmarking datasets and tools, such as the MMLU dataset (Hendrycks et al., 2021), HellaSwag (Zellers et al., 2019) and the Language Model Evaluation Harness framework (Gao et al., 2024), which are available under the MIT license.”
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Koepke, A. S., Zverev, D., Ginosar, S., & Efros, A. A. (2026) · arXiv · doi:10.48550/arxiv.2604.18572
“This claim is evaluated using three proxies for language performance: HellaSwag [106], GSM8K [12], and (1 − bitsperbyte ) .”
InfluentialOn Effectiveness and Efficiency of Agentic Tool-calling and RL Training
Liu, T., Qian, C., Cief, M., He, Y., Dan, D., Aletras, N., & Kazai, G. (2026) · arXiv
“To check whether tool-calling RL impacts general capabilities, we evaluate the base and RL-trained models on HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), and WinoGrande (Sakaguchi et al., 2021) using the official lm-evaluation-harness 3 (Gao et…”
Continual Learning via Sparse Memory Finetuning
Lin, J., Zettlemoyer, L., Ghosh, G., Yih, W.-T., Markosyan, A., Berges, V.-P., & Oğuz, B. (2025) · arXiv · doi:10.48550/arxiv.2510.15103 · Cited by 33
“For held-out performance, we measure F1 score on NaturalQuestions (Kwiatkowski et al., 2019) and accuracy on HellaSwag (Zellers et al., 2019).”
Learning What to Remember: Test-Time Training via Context Distillation
Wang, Z., Dang, X., Zhu, R.-J., Wen, Z., Fu, H., Chai, W., & Lee, J. D. (2026) · arXiv · doi:10.48550/arxiv.2608.01672
“For the 760M model, we further evaluate general downstream capabilities on four commonsense and knowledge benchmarks, including HellaSwag [54], ARC-Easy and ARC-Challenge [9], and MMLU [23].”
ATLAS: Learning to Optimally Memorize the Context at Test Time
Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y., Zhong, P., Razaviyayn, M., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2505.23735
MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging
Wang, J., Tian, C., Chen, K., Liu, Z., Mao, J., Zhao, W. X., Zhang, Z., & Zhou, J. (2026) · arXiv · doi:10.48550/arxiv.2601.17858
“…et al., 2024), OpenBookQA (Mihaylov et al., 2018), BBH (Suzgun et al., 2023), WorldSense (Hong et al., 2025), PIQA (Bisk et al., 2020), hellaswag (Zellers et al., 2019) and KOR-Bench (Ma et al., 2025)); (b) language understanding (race (Lai et al., 2017), SQuAD 2.0 (Rajpurkar et al., 2018),…”
InfluentialImplicit Language Models are RNNs: Balancing Parallelization and Expressivity
Schöne, M., Rahmani, B., Kremer, H., Falck, F., Ballani, H., & Gladrow, J. (2025) · arXiv · doi:10.48550/arxiv.2502.07827
“While this scheduling has also been shown to be compute-optimal (H¨agele et al., 2024) Downstream Task Details We evaluated our models as well as the baseline models on seven tasks: L AMBADA OPENAI (Paperno et al., 2016), H ELLASWAG (Zellers et al., 2019), P IQA (Bisk et al., 2020), A RC - EASY and A RC - CHALLENGE (C…”
Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Heineman, D., Hofmann, V., Magnusson, I., Gu, Y., Smith, N. A., Hajishirzi, H., Lo, K., & Dodge, J. (2025) · arXiv · doi:10.48550/arxiv.2508.13144
“MMLU [23], ARC [11], BoolQ [10], CSQA [59], OBQA [39], PiQA [4], SocialIQA [53], HellaSwag [69], WinoGrande [52], DROP [16], CoQA [48], Jeopardy [61], NaturalQs [28], SQuAD [47], TriviaQA [26], MedMCQA [43], MMLU Pro [64], AGI Eval [70], GPQA [49] Math.”
EntEval: A Holistic Evaluation Benchmark for Entity RepresentationsPreprint
Chen, M., Chu, Z., Chen, Y., Stratos, K., & Gimpel, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00137
A Critical Look at Benchmarking Datasets: Problem of finding relationship between sentences
Amasyalı, M. F., Demirci, D., & Uslu, M. N. (2019) · Innovations in Intelligent Systems and Applications Conference (ASYU) · doi:10.1109/asyu48272.2019.8946418
A multi-faceted approach to document quality assessment
Shen, A. (2020) · Minerva Access (University of Melbourne)
Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks
Iazykova, T., Быстрова, О. А., Kapelyushnik, D., & Kutuzov, A. (2021) · Computational Linguistics and Intellectual Technologies · doi:10.28995/2075-7182-2021-20-302-317
Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense TasksPreprint
Chang, T.-Y., Liu, Y., Gopalakrishnan, K., Hedayatnia, B., Zhou, P., & Hakkani-Tür, D. Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05457 · Cited by 2
ExperienceGen 1.0: A Text Generation Challenge Which Requires Deduction and Induction Ability
Zhang, H., Liu, P., Yu, D., & Zhang, S. (2021) · Natural Language Processing and Chinese Computing · doi:10.1007/978-3-030-88483-3_2
Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning
Utama, P. A., Moosavi, N. S., Sanh, V., & Gurevych, I. (2021) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.48550/arxiv.2109.04144
Fine-Tuned Transformers Show Clusters of Similar Representations Across LayersPreprint
Phang, J., Liu, H., & Bowman, S. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08406 · Cited by 9
From knowledge augmentation to multi-tasking: towards human-like dialogue systems
Yang, T. H. (2023) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/168465
Generation of Arabic Commonsense Explanations
Beheitt, M. E. G., & HajHmida, M. B. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-42430-4_43
Intermediate Task Ensembling for Sarcasm Detection
Jinks, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47994-6_2
Building a Commonsense Inference Dataset based on Basic Events and its Application
Omura, K., Kawahara, D., & Kurohashi, S. (2023) · Journal of Natural Language Processing · doi:10.5715/jnlp.30.1206
A Large Financial Chat ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-lsc94
Tasks for LLMs and Their Evaluation
Ostapuk, N., & Audiffren, J. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_6 · Cited by 2
Towards Flow Graph Prediction of Open-Domain Procedural Texts
Shirai, K., Kameko, H., & Mori, S. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.479
A Novel Lightweight Language Model Architecture with Flexible Parameters
Chakraborty, S. (2024) · Communications on Applied Nonlinear Analysis · doi:10.52783/cana.v31.786
“As the field of NLP continues [14] to advance, SwiFTeDLM stands as a testament to the potential for innovation in addressing the delicate balance between computational efficiency and linguistic proficiency. https://internationalpubls.com In future work, it would be pertinent to explore additional…”
BackgroundRecognition of Mental and Physical Adjectives in An Efficient and Automatic Style
Yang, F. (2024) · doi:10.1109/icnlp60986.2024.10692615
“Reasoning between physical events are studied in commonsense question answering task [34], [37].”
BackgroundEndriss, U., Melo, F. S., Bach, K., Bugarín-Diz, A., Alonso, J. M., Barro, S., & Heintz, F. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia392
“We perform few-shot task classification to evaluate the fine-tuned models using lm-evaluation-harness [16] strategy on common sense reasoning datasets: PIQA [3], HellaSwag [49], BoolQ [8], WinoGrande [34], ARC-easy [9], ARC-challenge [9] and OpenbookQA [30].”
Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in TransformersPreprint
He, S., Ge, T., Sun, G., Tian, B., Wang, X., Li, A., & Yu, D. (2024) · Qeios · doi:10.32388/kyqecv
G2DiaR: Enhancing Commonsense Reasoning of LLMs with Graph-to-Dialogue & Reasoning
Lian, H., Chen, K., Sun, Z., Gao, Y., & Li, Y. (2024) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp63861.2024.10800655
Fine-Tuning PHI-3 for Multiple-Choice Question Answering: Methodology, Results, and ChallengesPreprint
Hisham, M. (2025) · Qeios · doi:10.32388/z95x8o
Capturing Bias Diversity in LLMs
Gosavi, P. P., Kulkarni, V. M., & Smeaton, A. F. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852459
“The first is by identifying and using correlations and causal relationships between the protected attributes and other data features while the second is caused by under-representation of minority groups in the training data [14].”
Enhancing Small Language Models for Graph Tasks Through Graph Encoder Integration
Oh, D., Kang, S., Kim, H.-J., & Oh, D. (2025) · Applied Sciences · doi:10.3390/app15052418
Barzizza, E., Biasetton, N., Caligiuri, G., Pennisi, D., & Salmaso, L. (2025) · Journal of Machine Intelligence and Data Science · doi:10.11159/jmids.2025.003
LEP: Leveraging Local Entropy Pruning for Sparsity in Large Language Models
Chen, Y., Cheng, B. C., Zhang, Y., Zhang, S., Wu, Z., & Meng, F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888911
“Additionally, we evaluate the zero-shot accuracy on common sense benchmarks 2 such as RTE [32], HellaS [33], MNLI [34], and OBQA [35].”
Cache Partitioning for Mitigating Timing Side-Channel Attacks in LLM Serving Systems
Pang, Z., Wang, W., & Liao, Y. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913329
“…reasoning; MMLU [2], the Massive Multitask Language Understanding dataset that covers a wide range of disciplines including science, humanities, and social sciences; and Hellaswag [18], a commonsense reasoning dataset containing a large number of texts requiring commonsense judgment and reasoning.”
DCFT: Dependency-aware continual learning fine-tuning for sparse LLMs
Wang, Y., Wang, Y., & Yin, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129897
WayOutBench: Evaluating Multimodal LLMs in Interactive Environments
Margaritis, D., Karidas, D., Papadopoulos, P. G., & Leligou, H. C. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-87647-9_13
Zanuz, L., & Rigo, S. J. (2025) · RECIMA21 - Revista Científica Multidisciplinar · doi:10.47820/recima21.v6i5.6426
Automated Pruning Framework for Large Language Models Using Combinatorial Optimization
Ratsapa, P., Thonglek, K., Chantrapornchai, C., & Ichikawa, K. (2025) · AI · doi:10.3390/ai6050096
“In our experiments, we tested against the HellaSwag dataset [41], a widely recognized benchmark for common-sense reasoning tasks.”
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon LayersPreprint
Allen-Zhu, Z., & Alfarano, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5240330
Pugh, S., Ruchkin, I., Weimer, J. E., & Lee, I. (2025) · ACM Transactions on Cyber-Physical Systems · doi:10.1145/3734695 · Cited by 1
Reconstruct the Pruned Model Without Retraining
Wang, P., Fan, Z., Hu, S., Chen, Z., Wang, Y., & Wang, Y. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3568224 · Cited by 4
“We conduct experiments on seven pivotal common sense benchmarks to evaluate the model’s zero-shot performance: BoolQ [37], PIQA [38], HellaSwag [39], WinoGrande [40], ARC-easy [41], ARC-challenge [42] and OpenbookQA [42].”
Integer Unit-Based Outlier-Aware LLM Accelerator Preserving Numerical Accuracy of FP-FP GEMM
Lee, J., & Kim, J. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10992868
Hennekeuser, D., Vaziri, D. D., Golchinfar, D., & Stevens, G. (2025) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-025-01169-5
“Die meisten bekannten Benchmarks für LLMs sind in Englischer Sprache und können daher keine Fähigkeiten in verschiedenen Sprachen bewerten (Clark et al. 2018; Zellers et al. 2019; Hendrycks et al. 2021; Sakaguchi et al. 2021; Lin et al. 2022; Zheng et al. 2023).”
Enhancing logical reasoning in language models: An investigation of the Capybara dataset
Guerrero, L. E. M., Ceballos, Y. F., & Rojas, L. D. T. (2025) · Contemporary Educational Technology · doi:10.30935/cedtech/16425
Wei, Y., & Li, L. (2024) · International Conference on Information Science and Education (ICISE-IE) · doi:10.1109/icise-ie64355.2024.11025434
Advancing Large Language Models in Code Generation: Usaco Benchmark and Bug Mitigation Insights
Trentini, J., Liu, V., Peng, Y., & Zong, Z. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00057 · Cited by 1
“These benchmarks include, but are not limited to, ARC [52], HellaSwag [53], MMLU [54], Truth-fulQA [55], WinoGrade [56], and GSM8K [57].”
Exploring and Enhancing Advanced MoE Models: From Deepspeed-MoE to DeepSeek-V3
Zhang, Z., Gao, Y., & Chen, D. (2025) · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT) · doi:10.1109/ainit65432.2025.11035928
Beyond the Benchmark: A Customizable Platform for Real-Time, Preference-Driven LLM Evaluation
Zografos, G., & Moussiades, L. (2025) · Electronics · doi:10.3390/electronics14132577 · Cited by 2
A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning
Xu, J., Diao, B., Qi, C., Zhao, S., Wang, R., & Xu, Y. (2025) · IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) · doi:10.1109/ccgrid64434.2025.00059
“To validate the effectiveness of our approach, we conducted evaluations across eight datasets: BoolQ [16], PIQA [17], SIQA [18], Winogrande [19], ARC-Easy [20], ARC-Challenge [20], OBQA [21], and HellaSwag [22].”
A Roadmap for Alignable Algorithmic Decision-Makers in the Medical Triage Domain
Hu, B., Chan, D., Sorensen, T., Chen, X., Ji, H., Choi, Y., Darrell, T., & Basharat, A. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00203
Self-citationSpicing Up LLMs: The Role of PAPRICA Pruning in Linear Computation Coding
Fröhlich, J. S., Rosenberger, H., & Müller, R. R. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00064
“The datasets are the following: Winogrande [25], HellaSwag [26], RTE [27], BoolQ [28], ARC Easy and ARC Challenge [29] as well as OpenBookQA [30].”
Zhao, C., Zeng, B., Chen, K., & Lin, X. A. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9994-0_39
Probing Fine-Grained Hierarchical Concept Comprehension and Generation in Large Language Models
Sun, K., Bai, Y., Tu, S., Li, J., & Hou, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3592339
IterSelectTune: An Iterative Data Selection Framework for Efficient Instruction Tuning
Song, J., Liu, S., Zhu, B., & Rao, Y. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-0014-7_28 · Cited by 8
Preaching to the ChoIR: Lessons IR Should Share with AI
Demartini, G., Hauff, C., Lease, M., Mizzaro, S., Roitero, K., Sanderson, M., Scholer, F., Shah, C. S., Spina, D., Thomas, P., et al. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744612
“Common benchmark examples include ARC-C [32], CRASS [46], HellaSwag [178], MMLU [59], OpenBookQA [90], RACE [73], and SciQ [171].”
Matryoshka Model Learning for Improved Elastic Student Models
Verma, C. K., Timmaraju, A. S., Hsieh, C., Damle, S., Bui, N., Zhang, Y., Chen, W., Liu, X., Jain, P., & Dhillon, I. S. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737245
“We train a decoder model with GPT-2 Medium [37] architecture from scratch on 100B tokens from C4 [38] corpus.”
Advanced NLP Techniques for Enhancing ChatGPT Performance
Roy, B., Roy, P., Ghosh, S., Paul, T., & Paul, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-9841-8.ch001
A cyclical loss-based optimization algorithm for pretraining LLMs on noisy data
Kesgin, H. T., & Amasyalı, M. F. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114189 · Cited by 2
On Semantic Association Capabilities of GPT LLMs in a Game of Word Associations
Kondić, A. Z., & Milovanović, V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-99201-8_4
ArabianGPT: An Arabic Language Adaptation of GPT-2
Koubâa, A., Ammar, A., Ghouti, L., Nacar, O., & Sibaee, S. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-90573-5_2
Creating a Large Clean Web Corpus for Turkish
Uzun, M. E., Erdem, Y., İzdaş, T., Sancak, Ö., Zeer, A., İnce, E., Uz, A., Shbib, O., Doğan, E., Kesgin, H. T., et al. (2025) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi65310.2025.11141276
DBC: Drift-aware Binary Code for Drift-tolerant Deep Neural Networks
Choi, I., Chung, J., & Yang, J.-S. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132851
“2 1B [3] and Gemma-2 2B [2] are evaluated using the HellaSwag benchmark [25].”
ICT-QA: Question Answering Over Multi-Modal Contexts Including Image, Chart, and Text Modalities
Jang, Y., Kong, H., Kim, G., Lee, Y., Choi, J., & Bae, K. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00019
A Cross-model Fusion-aware Framework for Optimizing (gather-matmul-scatter)s Workload
Lian, Y., Gou, Z., Han, Y., Yu, Z., Xu, J., Yuan, S., Pei, Z., Zhang, X., Xu, N., & Dai, G. (2025) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1109/dac63849.2025.11132773
“[14], [30].”
Weight-Sharing NAS with Architecture-Agnostic Intermediate Representation
Yu, S., Mazaheri, A., & Jannesari, A. (2025) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3731545.3731587
“2-3B, we evaluate the compressed models on three distinct tasks: CommonsenseQA [45], HellaSwag [59], and Truth-fulQA MC2 [31].”
Coded Deep Learning: Framework and Algorithm
Yang, E., & Hamidi, S. M. (2025) · IEEE Transactions on Information Theory · doi:10.1109/tit.2025.3610095 · Cited by 1
“• Datasets and Settings: To evaluate the generalization ability of our quantized models, we perform zero-shot evaluation on two representative benchmarks: BoolQ [74] and HellaSwag [75].”
QSLR: Post-Training Compression via Quantized Sparse and Low-Rank Factorization
Kwon, E. (2025) · IEEE Access · doi:10.1109/access.2025.3615473
Liu, A., Qi, B., & Hu, X. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02899-0_6
Jung, I.-J., Yang, G., Min, J., & Kim, J.-Y. (2025) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3725843.3756115 · Cited by 5
“…based on FP16 precision, both PipeFlash and PipeSSD have confirmed that there is no accuracy loss compared to conventional FAs and SSD across eight benchmarks (wikitext-2 [30], Winogrande [46], ARC-challenge and ARC-easy [8], LAMBADA-openai [42], PIQA [6], OpenBookQA [31], and HellaSwag [54]).”
TALE: Token-Adaptive Low-Rank KVCache Approximation with Reconstruction Elimination
Lee, J., Hwang, S.-W., Qiao, A., Campos, D., Yao, Z., & He, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.39
Bridging the Gap in Less-Resourced Languages: Building a Benchmark for Kyrgyz Language Models
Turatali, T., Turdubaeva, A., Zhenishbekov, I., Suranbaev, Z., Alekseev, A., & Izmailov, R. (2025) · doi:10.1109/ubmk67458.2025.11206960
“Specifically, we selected: • Commonsense reasoning & understanding: HellaSwag [7] and WinoGrande [6].”
Cho, H., Padhy, A. P., Camacho, F., & Mukhopadhyay, S. (2025) · IEEE Access · doi:10.1109/access.2025.3625771 · Cited by 2
Expertfuse: A huffman tree-based gradual expert integration framework for MoE models
Fang, Y.-Z., & Huang, J.-D. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.108274
Yan, J., Luo, Y., & Zhang, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3626926
“Inspired by [40], we use LLM-as-a-Judge to rate 1-5 for each refuted response as well as the explanation for the rating.”
ScaleNet: Scaling up Pretrained Neural Networks With Incremental Parameters
Hao, Z., Guo, J., Shen, L., Han, K., Tang, Y., Hu, H., & Wang, Y. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3624968
“Zero-shot performance on common-sense reasoning benchmarks is used for evaluation, including BoolQ [63], PIQA [64], HellaSwag [65], WinoGrande [66], ARC-easy [67], ARC-challenge [67], and OpenbookQA [68].”
Enabling Unstructured Sparse Fine-Tuning and Inference for Foundation Models on Wafer-Scale Engine
Zheng, H., Zeng, Y., Song, L., Emani, M. K., & Dong, W. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3731599.3767395
Constraint Back-translation Improves Complex Instruction Following of Large Language Models
Qi, Y., Peng, H. Y., Wang, X., Xu, B., Hou, L., & Li, J. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761324 · Cited by 20
Efficient Knowledge Transfer from Large to Small Language Models via Low-Overhead Query Mechanism
Ahemad, F. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761417 · Cited by 1
EFT-LR: Benchmarking Learning Rate Policies in Parameter-Efficient Large Language Model Fine-tuning
Jawad, M. T., & Wu, Y. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761630
“…through inter-personal interactions, (3) CommonsenseQA [43], which tests general commonsense knowledge via multiple-choice questions, (4) Hel-laSwag [54], which is a sentence completion task requiring nuanced commonsense reasoning, (5) OpenBookQA [30], which is a question-answering task…”
Optimización de ChatGPT para la clasificación de noticias falsas mediante FineTuning
Alcívar, C., Tomalá, D. F. R., & Bernabé, M. D. C. T. (2025) · Revista Científica Multidisciplinar G-nerando · doi:10.60100/rcmg.v6i2.816
“Sin embargo, su capacidad para clasificar noticias falsas de manera precisa y eficiente requiere una optimización específica mediante técnicas avanzadas de aprendizaje automático, como el fine-tuning (Zellers et al., 2019; Alonso & Fernández, 2023).”
Learnable Cloud-Guided LLM Quantization for Resource-Constrained Edge Devices
Deng, Q., Pang, T., Zhang, B., Nie, B., He, X., Mao, Y., & Wu, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-10459-5_33
Mooe: Optimal Expert Deployment Scheme in Low-Rank Adaptation architecture
Sun, S., Zhang, J., Ma, X., & Zhang, P. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227782
Metadata
- Authors
- Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4791-4800
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2946609015
- Cited by
- 830
- References
- 20
- Citations by year
- 2026: 200 · 2025: 263 · 2024: 109 · 2023: 93 · 2022: 50 · 2021: 61 · 2020: 44 · 2019: 10
- FWCI
- 15.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation (1703166, 1256082, 1637479, 1524371)
- Nvidia
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity
- Army Research Office (W911NF-15-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- commonsense reasoning, natural language inference, AI-generated text, BERT, benchmark design, pre-trained language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
