CoChat
Sign Up
PreprintCited by 13Topic ModelingNatural Language Processing TechniquesAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 1 of the 5 references it cites were found in a registry or an index, and 4 are not checked yet.

Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers

Open access
Pacchiardi, Lorenzo, Tesic, Marko, Cheke, Lucy G., Hernández-Orallo, José·2024·arXiv
DOI 10.48550/arxiv.2410.11672arXiv 2410.11672
Open this paper in CoChat — free accountOpen PDF

Abstract

The integrity of AI benchmarks is fundamental to accurately assess the capabilities of AI systems. The internal validity of these benchmarks - i.e., making sure they are free from confounding factors - is crucial for ensuring that they are measuring what they are designed to measure. In this paper, we explore a key issue related to internal validity: the possibility that AI systems can solve benchmarks in unintended ways, bypassing the capability being tested. This phenomenon, widely known in human and animal experiments, is often referred to as the 'Clever Hans' effect, where tasks are solved using spurious cues, often involving much simpler processes than those putatively assessed. Previous research suggests that language models can exhibit this behaviour as well. In several older Natural Language Processing (NLP) benchmarks, individual $n$-grams like "not" have been found to be highly predictive of the correct labels, and supervised NLP models have been shown to exploit these patterns. In this work, we investigate the extent to which simple $n$-grams extracted from benchmark instances can be combined to predict labels in modern multiple-choice benchmarks designed for LLMs, and whether LLMs might be using such $n$-gram patterns to solve these benchmarks. We show how simple classifiers trained on these $n$-grams can achieve high scores on several benchmarks, despite lacking the capabilities being tested. Additionally, we provide evidence that modern LLMs might be using these superficial patterns to solve benchmarks. This suggests that the internal validity of these benchmarks may be compromised and caution should be exercised when interpreting LLM performance results on them.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2410.11672

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A recurring concern with benchmarking is that models may use heuristics or shallow strategies to perform a task correctly without necessarily using the ability that is being tested.

    “A recurring concern with benchmarking is that models may use heuristics or shallow strategies to perform a task correctly without necessarily using the ability that is being tested [93,94].”

    From Re-evaluating Theory of Mind evaluation in large language models (2025) · cites this paper as [94]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 54

  1. Lasha Abzianidze, Joost Zwarts, and Yoad Winter. SpaceNLI: Evaluating the consistency of predicting inferences in space. ArXiv, abs/2307.02269, 2023. URL https://api. semanticscholar.org/CorpusID:259341771. · arXiv
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  3. Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsub- aie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781, 2024. · arXiv
  4. Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, and Mitsuhiro Okada. Eval- uating large language models with NeuBAROCO: Syllogistic reasoning ability and human-like biases, 2023. URL https://arxiv.org/abs/2306.12567. · arXiv
  5. Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995.
  6. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sid Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wil- son Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, Franc- cois Yvon, and Andy Zou. Lessons from the trenches on reproducible evaluation of language models. ArXiv, abs/2405.14782, 2024. URL https://api.semanticscholar.org/ CorpusID:269982020. · arXiv
  7. Samuel R. Bowman and George E. Dahl. What will it take to fix benchmarking in natural language understanding? ArXiv, abs/2104.02145, 2021. URL https://api.semanticscholar. org/CorpusID:233033916. · arXiv
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901.
  9. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  10. Boxi Cao, Mengjie Ren, Hongyu Lin, Xianpei Han, Feng Zhang, Junfeng Zhan, and Le Sun. StructEval: Deepen and broaden large language model assessment via structured evaluation, 2024. URL https://arxiv.org/abs/2408.03281. · arXiv
  11. Nigel H. Collier, Fangyu Liu, and Ehsan Shareghi. On reality and the limits of language data, 2022. URL https://arxiv.org/abs/2208.11981. · arXiv
  12. Dhairya Dalal, Paul Buitelaar, and Mihael Arcan. Calm-bench: A multi-task benchmark for eval- uating causality-aware language models. In Findings of the Association for Computational Lin- guistics: EACL 2023, pp. 296-311, 2023.
  13. Ernest Davis. Benchmarks for automated commonsense reasoning: A survey. ACM Computing Surveys, 56:1 -41, 2023. URL https://api.semanticscholar.org/CorpusID: 256697203.
  14. Ricardo Dominguez-Olmedo, Florian E Dorner, and Moritz Hardt. Training on the test task con- founds evaluation and emergence. arXiv preprint arXiv:2407.07890, 2024. · arXiv
  15. Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Communications of the ACM, 67(1):110-120, 2023.
  16. Dan Friedman, Alexander Wettig, and Danqi Chen. Finding dataset shortcuts with grammar induc- tion. arXiv preprint arXiv:2210.11560, 2022. · arXiv
  17. Lawrence M Friedman, Curt Furberg, David L DeMets, David M Reboussin, Christopher B Granger, et al. Fundamentals of clinical trials, volume 4. Springer, 2010.
  18. Anna Gábor, Noémi Kaszás, Tamás Faragó, Paula Pérez Fraga, Melinda Lovas, and Attila Andics. The acoustic bases of human voice identity processing in dogs. Animal Cognition, 25(4):905-916, 2022.
  19. Matt Gardner, William Merrill, Jesse Dodge, Matthew E Peters, Alexis Ross, Sameer Singh, and Noah A Smith. Competency problems: On finding and removing artifacts in language data. arXiv preprint arXiv:2104.08646, 2021. · arXiv
  20. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Hag- gai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. LegalBench: A collabora- tively built benchmark for measuring legal reasoning in large language models, 2023. URL https://arxiv.org/abs/2308.11462. · arXiv
  21. Jose Hernandez-Orallo. Gazing into clever hans machines. Nature Machine Intelligence, 1(4): 172-173, 2019.
  22. Jennifer Hu and Michael C. Frank. Auxiliary task demands mask the capabilities of smaller lan- guage models. ArXiv, abs/2404.02418, 2024. URL https://api.semanticscholar. org/CorpusID:268876128. · arXiv
  23. Anna A. Ivanova. Running cognitive evaluations on large language models: The do's and the don'ts. ArXiv, abs/2312.01276, 2023. URL https://api.semanticscholar.org/ CorpusID:265610018. · arXiv
  24. Minhao Jiang, Ken Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Does data contamination make a difference? insights from intentionally contaminating pre- training data for language models. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
  25. Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fer- nando Gonzalez, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. CLad- der: Assessing causal reasoning in language models. In NeurIPS, 2023. URL https: //openreview.net/forum?id=e2wtjx0Yqu.
  26. Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Ken- taro Inui. When choosing plausible alternatives, Clever Hans can be clever. arXiv preprint arXiv:1911.00225, 2019. · arXiv
  27. Pride Kavumba, Ryo Takahashi, and Yusuke Oda. Are prompt-based models clueless? In An- nual Meeting of the Association for Computational Linguistics, 2022. URL https://api. semanticscholar.org/CorpusID:248780567.
  28. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. · arXiv
  29. Thomas Liao, Rohan Taori, Deborah Raji, and Ludwig Schmidt. Are we learning yet? a meta review of evaluation failures across machine learning. In J. Van- schoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https: //datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/ 2021/file/757b505cfd34c64c85ca5b5690ee5293-Paper-round2.pdf.
  30. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. Wanli: Worker and ai collabora- tion for natural language inference dataset creation. arXiv preprint arXiv:2201.05955, 2022. · arXiv
  31. Man Luo, Shrinidhi Kumbhar, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, Chitta Baral, et al. Towards logiglue: A brief survey and a benchmark for analyzing logical reasoning capabilities of language models. arXiv preprint arXiv:2310.00836, 2023. · arXiv
  32. HaDi MaBouDi, Andrew B Barron, Sun Li, Maria Honkanen, Olli J Loukola, Fei Peng, Wenfeng Li, James AR Marshall, Alex Cope, Eleni Vasilaki, et al. Non-numerical strategies used by bees to solve numerical cognition tasks. Proceedings of the Royal Society B, 288(1945):20202711, 2021.
  33. Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282, 2012.
  34. Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N. Halgamuge. In- adequacies of large language model benchmarks in the era of generative artificial intelli- gence. ArXiv, abs/2402.09880, 2024. URL https://api.semanticscholar.org/ CorpusID:267682172. · arXiv
  35. Eduardo Mercado III, Deirdre A Killebrew, Adam A Pack, Inés VB Mácha, and Louis M Herman. Generalization of 'same-different'classification abilities in bottlenosed dolphins. Behavioural Processes, 50(2-3):79-94, 2000.
  36. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Compu- tational Linguistics, 12:933-949, 2024.
  37. Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with CogEval. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=VtkGvGcGe3.
  38. Yael Moros Daval. Automated annotation of meta-features for predicting language model per- formance in natural language processing tasks, 2023. URL http://hdl.handle.net/ 10251/196727.
  39. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. ArXiv, abs/1910.14599, 2019. URL https://api.semanticscholar.org/CorpusID:207756753. · arXiv
  40. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042, 2018. · arXiv
  41. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
  42. Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. Data contamination through the lens of time. arXiv preprint arXiv:2310.10628, 2023. · arXiv
  43. Danaja Rutar, Lucy Gaia Cheke, José Hernández-Orallo, Alva Markelius, and Wout Schellaert. General interaction battery: Simple object navigation and affordances (GIBSONA). Available at SSRN 4924246, 2024.
  44. Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=qFVVBzXxR2V.
  45. Chenglei Si, Shuohang Wang, Min-Yen Kan, and Jing Jiang. What does BERT learn from multiple- choice reading comprehension datasets? arXiv preprint arXiv:1910.12391, 2019. · arXiv
  46. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. · arXiv
  47. Bob L Sturm. A simple method to determine if a music information retrieval system is a "horse". IEEE Transactions on Multimedia, 16(6):1636-1644, 2014.
  48. Alon Talmor, Ori Yoran, Ronan Le Bras, Chandrasekhar Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. CommonsenseQA 2.0: Exposing the limits of AI through gami- fication. ArXiv, abs/2201.05320, 2021. URL https://api.semanticscholar.org/ CorpusID:237263476. · arXiv
  49. Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, Yue Zhang, et al. Glore: Evaluating logical reason- ing of large language models. arXiv preprint arXiv:2310.09107, 2023. · arXiv
  50. Murayama Tsukasa, Yuki Fujii, Takayuki Hashimoto, Aya Shimoda, So Iijima, Kohei Hayasaka, Narumi Shiroma, Mana Koshikawa, Hiroshi Katsumata, Makoto Soichi, et al. Preliminary study of object labeling using sound production in a beluga. International Journal of Comparative Psychology, 25(3), 2012.
  51. Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computa- tional Linguistics, 8:621-633, 2020.
  52. Paul Verhaeghen. Aging and vocabulary score: A meta-analysis. Psychology and aging, 18(2):332, 2003.
  53. Yuqing Wang and Yun Zhao. Rupbench: Benchmarking reasoning under perturbations for robust- ness evaluation in large language models. arXiv preprint arXiv:2406.11020, 2024. · arXiv
  54. Linying Yang, Oscar Clivio, Vik Shirvaikar, and Fabian Falck. A critical review of causal infer- ence benchmarks for large language models. In AAAI 2024 Workshop on "Are Large Language Models Simply Causal Parrots?", 2023. URL https://openreview.net/forum?id= mRwgczYZFJ.

Cited by

Sorted by citations

Related

Metadata

Authors
Pacchiardi, L., Tesic, M., Cheke, L. G., & Hernández-Orallo, J.
Affiliations
University of Cambridge (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2410.11672
arXiv
2410.11672
OpenAlex
W4403575061
Cited by
1
References
0
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
Clever Hans effect, internal validity, large language models, n-gram features

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review