CoChat
Sign Up
PreprintCited by 8Topic ModelingNatural Language Processing TechniquesLanguage Development and Disorders

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

Auxiliary task demands mask the capabilities of smaller language models

Open access
Hu, Jennifer, Frank, Michael C.·2024·arXiv
DOI 10.48550/arxiv.2404.02418arXiv 2404.02418
Open this paper in CoChat — free accountOpen PDF

Abstract

Developmental psychologists have argued about when cognitive capacities such as language understanding or theory of mind emerge. These debates often hinge on the concept of "task demands" -- the auxiliary challenges associated with performing a particular evaluation -- that may mask the child's underlying ability. The same issues arise when measuring the capacities of language models (LMs): performance on a task is a function of the model's underlying knowledge, combined with the model's ability to interpret and perform the task given its available resources. Here, we show that for analogical reasoning, reflective reasoning, word prediction, and grammaticality judgments, evaluation methods with greater task demands yield lower performance than evaluations with reduced demands. This "demand gap" is most pronounced for models with fewer parameters and less training data. Our results illustrate that LM performance should not be interpreted as a direct indication of intelligence (or lack thereof), but as a reflection of capacities seen through the lens of researchers' design choices.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.02418

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Evaluating LLMs using adversarially constructed examples introduces complexities that may shift the target of evaluation away from 'pure' ToM and towards more general reasoning capacities.

    “Also, evaluating LLMs using adversarially constructed examples introduces complexities that may shift the target of evaluation away from 'pure' ToM and towards more general reasoning capacities [55].”

    From Re-evaluating Theory of Mind evaluation in large language models (2025) · cites this paper as [55]

    The Background and related work section says auxiliary task demands can make an evaluation less ‘pure,’ and the Discussion advises considering how such demands interact with adversarial evaluation goals. The paper supports the general validity concern, but it does not specifically show that adversarially constructed theory-of-mind examples shift evaluation toward general reasoning capacities. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • As test items become more adversarial and complex, we run the risk of no longer primarily testing ToM abilities, but instead introducing unintended tests of the ability of models to overcome auxiliary demands associated with the task.

    “As this happens, we run the risk of no longer primarily testing ToM abilities, but instead introducing unintended tests of the ability of models to overcome auxiliary demands associated with the task (e.g. longer context windows, keeping track of more agents, keeping track of unfamiliar vocabulary items or performing physical reasoning) [55].”

    From Re-evaluating Theory of Mind evaluation in large language models (2025) · cites this paper as [55]

    The Background and related work section says auxiliary demands can make an evaluation a less pure measure of the capacity it targets, supporting the general warning. The paper mentions theory of mind in its Abstract and Introduction, but its reported evaluations do not test theory of mind; nor do the cited passages establish the sentence’s specific examples of longer contexts, more agents, unfamiliar vocabulary, or physical reasoning as auxiliary demands in theory-of-mind tests. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Hu & Frank (2024) found that direct probability estimation yields results that differ from and are often better than the metalinguistic approach.

    “Across model sizes and datasets, Hu and Frank find that direct probability estimation yields results that differ from and are often better than the metalinguistic approach.”

    From Anthropocentric Bias in Language Model Evaluation (2026) · cites this paper as Hu & Frank (2024)

  • Metalinguistic prompting introduces an auxiliary task demand-the ability to generate explicit grammaticality judgments-that is irrelevant to the underlying syntactic competence of interest.

    “They conclude that metalinguistic prompting introduces an auxiliary task demand - the ability to generate explicit grammaticality judgments - that is irrelevant to the underlying syntactic competence of interest.”

    From Anthropocentric Bias in Language Model Evaluation (2026) · cites this paper as Hu & Frank (2024)

  • Direct probability estimation more validly measures the target capacity than metalinguistic prompting.

    “In contrast, direct probability estimation more validly measures the target capacity.”

    From Anthropocentric Bias in Language Model Evaluation (2026) · cites this paper as Hu & Frank (2024)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 48

  1. Elika Bergelson and Daniel Swingley. At 6-9 months, human infants know the meanings of many common nouns. Proceedings of the National Academy of Sciences, 109(9):3253-3258, February 2012. doi: 10.1073/pnas.1113380109. URL https://doi.org/10.1073/pnas. 1113380109. · DOI
  2. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling, 2023. URL https: //arxiv.org/abs/2304.01373.
  3. Samuel Bowman. The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7484-7499, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.516. URL https://aclanthology.org/2022.acl-long.516. · DOI
  4. Samuel R. Bowman and George Dahl. What Will it Take to Fix Benchmarking in Natural Language Understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4843-4855, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.naacl-main.385. URL https://aclanthology.org/2021.naacl-main.385. · DOI
  5. Peter Carruthers. Mindreading in infancy. Mind & Language, 28(2):141-172, 2013.
  6. Tyler A. Chang and Benjamin K. Bergen. Language Model Behavior: A Comprehensive Survey. Computational Linguistics, 50(1):293-350, March 2024. ISSN 0891-2017. doi: 10.1162/coli_a_00492. URL https://doi.org/10.1162/coli_a_00492. · DOI · Paper page
  7. Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological Bulletin, 52(4):281, 1955.
  8. Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51):e2309583120, December 2023a. doi: 10.1073/pnas.2309583120. URL https://doi.org/10.1073/pnas.2309583120. Publisher: Proceedings of the National Academy of Sciences. · DOI · Paper page
  9. Vittoria Dentella, Elliot Murphy, Gary Marcus, and Evelina Leivada. Testing AI performance on less frequent aspects of language reveals insensitivity to underlying meaning, 2023b. URL https://arxiv.org/abs/2302.12313. · arXiv
  10. Chaz Firestone. Performance vs. competence in human-machine comparisons. Proceedings of the National Academy of Sciences, 117(43):26562-26571, October 2020. URL https:// doi.org/10.1073/pnas.1905334117. Publisher: Proceedings of the National Academy of Sciences. · DOI · Paper page
  11. Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451-452, 2023a.
  12. Michael C Frank. Openly accessible llms can help us to understand human cognition. Nature Human Behaviour, 7(11):1825-1827, 2023b.
  13. Shane Frederick. Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4):25-42, 2005. doi: 10.1257/089533005775196732. URL https://www.aeaweb.org/ articles?id=10.1257/089533005775196732. · DOI · Paper page
  14. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette- Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu-hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open Models Based on Gemini Research and Technology, 2024. URL https://arxiv.org/abs/2403.08295. · arXiv
  15. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Py- atkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richard- son, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Han- naneh Hajishirzi. OLMo: Accelerating the Science of Language Models, 2024. URL https://arxiv.org/abs/2402.00838. · arXiv
  16. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in Chat- GPT. Nature Computational Science, 3(10):833-838, October 2023. ISSN 2662-8457. doi: 10.1038/s43588-023-00527-x. URL https://doi.org/10.1038/s43588-023-00527-x. · DOI · Paper page
  17. Petra Hendriks and Charlotte Koster. Production/comprehension asymmetries in language acquisition. Asymmetries in Language Acquisition, 120(8):1887-1897, August 2010. ISSN 0024-3841. doi: 10.1016/j.lingua.2010.02.002. URL https://www.sciencedirect.com/ science/article/pii/S0024384110000215. · DOI
  18. Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040-5060, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.emnlp-main.306. URL https://aclanthology.org/2023.emnlp-main.306. · DOI · Paper page
  19. Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language models align with human judgments on key grammatical constructions, 2024. URL https://arxiv.org/abs/2402.01676. · arXiv
  20. Anna A. Ivanova. Running cognitive evaluations on large language models: The do's and the don'ts, 2023. URL https://arxiv.org/abs/2312.01276. · arXiv
  21. Roni Katzir. Why Large Language Models Are Poor Theories of Human Linguistic Cognition: A Reply to Piantadosi. Biolinguistics, 17, 2023. URL https://doi.org/10.5964/bioling. 13153. · DOI
  22. Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna A. Ivanova. Comparing Plausibility Estimates in Base and Instruction-Tuned Large Language Models, 2024. URL https://arxiv.org/abs/2403.14859. · arXiv
  23. Rachel Keen. Representation of objects and events: Why do infants look so smart and toddlers look so dumb? Current Directions in Psychological Science, 12(3):79-83, 2003.
  24. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking Benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.324. URL https://aclanthology.org/2021.naacl-main.324. · DOI · Paper page
  25. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large Language Models are Zero-Shot Reasoners. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 22199-22213.
  26. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf.
  27. Andrew K Lampinen, Ishita Dasgupta, Stephanie C Y Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models, like humans, show content effects on reasoning tasks. PNAS Nexus, 3(7):pgae233, July 2024. ISSN 2752-6542. doi: 10.1093/pnasnexus/pgae233. URL https://doi.org/10. 1093/pnasnexus/pgae233. · DOI · Paper page
  28. Andrew Kyle Lampinen. Can language models handle recursively nested grammatical structures? A case study on comparing models and humans, 2023. URL https://arxiv. org/abs/2210.15303.
  29. Tal Linzen. How Can We Accelerate Progress Towards Human-like Linguistic Generaliza- tion? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, pp. 5210-5217, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.465. URL https://aclanthology.org/2020.acl-main.465. · DOI · Paper page
  30. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016. doi: 10.1162/tacl_a_00115. URL https://aclanthology.org/Q16-1037. Place: Cambridge, MA Publisher: MIT Press. · DOI
  31. Chenyang Lyu, Minghao Wu, and Alham Fikri Aji. Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models, 2024. URL https://arxiv.org/ abs/2402.13887. · arXiv
  32. Rebecca Marvin and Tal Linzen. Targeted Syntactic Evaluation of Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1151. URL https://aclanthology.org/D18-1151. · DOI · Paper page
  33. Kanishka Misra. minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models, 2022. URL https://arxiv.org/abs/2203.13112. · arXiv
  34. Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating Cognitive Maps and Plan- ning in Large Language Models with CogEval. In Thirty-seventh Conference on Neural In- formation Processing Systems, 2023. URL https://openreview.net/forum?id=VtkGvGcGe3.
  35. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/P16-1144. · DOI · Paper page
  36. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural In- formation Processing Systems Track on Datasets and Benchmarks, volume 1. Cur- ran, 2021. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/ paper/2021/file/084b6fbb10729ed4da8c3d3f5a3ae7c9-Paper-round2.pdf.
  37. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models, 2024. URL https://arxiv.org/abs/2402.16786. · arXiv
  38. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Commun. ACM, 64(9):99-106, August 2021. ISSN 0001-0782. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. Place: New York, NY, USA Publisher: Association for Computing Machinery. · DOI · Paper page
  39. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are Emergent Abilities of Large Language Models a Mirage? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=ITw9edRDlD.
  40. David Schlangen. Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research, 2019. URL https://arxiv.org/abs/1908.10747. · arXiv
  41. James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the Stepwise Nature of Self-Supervised Learning. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https://proceedings.mlr.press/ v202/simon23a/simon23a.pdf.
  42. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. URL https://arxiv.org/abs/2307.09288. · arXiv
  43. Polina Tsvilodub, Hening Wang, Sharon Grosch, and Michael Franke. Predictions from language models for multiple-choice tasks are not robust under variation of scoring methods, 2024. URL https://arxiv.org/abs/2403.00998. · arXiv
  44. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8, 2020. URL https://doi.org/ 10.1162/tacl_a_00321. Publisher: MIT Press. · DOI · Paper page
  45. Taylor Webb, Keith J. Holyoak, and Hongjing Lu. Emergent analogical reasoning in large lan- guage models. Nature Human Behaviour, 7(9):1526-1541, September 2023. ISSN 2397-3374. doi: 10.1038/s41562-023-01659-w. URL https://doi.org/10.1038/s41562-023-01659-w. · DOI · Paper page
  46. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abili- ties of Large Language Models. Transactions on Machine Learning Research, 2022a. URL https://openreview.net/forum?id=yzkSU5zdwD.
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL https://openreview.net/forum?id=_VjQlMeSB_J.
  48. Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi. The Generative AI Paradox: "What It Can Create, It May Not Understand". In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=CF8H8MS5P8.

Cited by

Sorted by citations

Metadata

Authors
Hu, J., & Frank, M. C.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2404.02418
arXiv
2404.02418
OpenAlex
W4393968405
Cited by
8
References
3
Citations by year
2026: 2 · 2025: 2 · 2024: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
Keywords
task demands, demand gap, language model evaluation, model size, parameter count, training data size, analogical reasoning, reflective thinking, word prediction, grammaticality judgment, intelligence testing, cognitive development, theory of mind, language comprehension

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review