CoChat
Sign Up
Cited by 69Logic, Reasoning, and KnowledgeTopic ModelingNatural Language Processing Techniques

Language models, like humans, show content effects on reasoning tasks

Open access
Lampinen, Andrew K, Dasgupta, Ishita, Chan, Stephanie C Y, Sheahan, Hannah R, Creswell, Antonia, Kumaran, Dharshan, McClelland, James L, Hill, Felix·2024·PNAS Nexus
DOI 10.1093/pnasnexus/pgae233PMC PMC11250216
Open this paper in CoChat — free account

Abstract

Abstract reasoning is a key ability for an intelligent system. Large language models (LMs) achieve above-chance performance on abstract reasoning tasks but exhibit many imperfections. However, human abstract reasoning is also imperfect. Human reasoning is affected by our real-world knowledge and beliefs, and shows notable “content effects”; humans reason more reliably when the semantic content of a problem supports the correct logical inferences. These content-entangled reasoning patterns are central to debates about the fundamental nature of human intelligence. Here, we investigate whether language models—whose prior expectations capture some aspects of human knowledge—similarly mix content into their answers to logic problems. We explored this question across three logical reasoning tasks: natural language inference, judging the logical validity of syllogisms, and the Wason selection task. We evaluate state of the art LMs, as well as humans, and find that the LMs reflect many of the same qualitative human patterns on these tasks—like humans, models answer more accurately when the semantic content of a task supports the logical inferences. These parallels are reflected in accuracy patterns, and in some lower-level features like the relationship between LM confidence over possible answers and human response times. However, in some cases the humans and models behave differently—particularly on the Wason task, where humans perform much worse than large models, and exhibit a distinct error pattern. Our findings have implications for understanding possible contributors to these human cognitive effects, as well as the factors that influence language model performance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/pnasnexus/pgae233

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using Prolog instead of natural language in evaluations avoids ambiguity and content effects that can impact performance, as models may draw on prior knowledge rather than the stated context.

    “we opt to use Prolog instead of natural language in our evaluations, in order to avoid ambiguity and content effects that can impact performance, as models may draw on prior knowledge rather than the stated context (Lampinen et al., 2024).”

    From Probing for Long-Horizon Deductive Reasoning Capabilities in Language Models with Prolog (2026) · cites this paper as Lampinen et al., 2024

    The Results and Discussion (pages unavailable) show that models’ reasoning performance is affected by whether problem content aligns with real-world expectations, supporting the concern about prior knowledge overriding a stated premise. The paper does not evaluate Prolog or establish that using it instead of natural language avoids ambiguity or content effects. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • LLMs often fail on informal reasoning tasks because they either lack human cognitive abilities or replicate human-like cognitive biases and heuristics.

    “These failures arise either because LLMs lack certain fundamental cognitive abilities possessed by humans - leading to errors that humans typically avoid (Han et al., 2024b) - or because LLMs replicate human-like cognitive biases and heuristics, resulting in analogous mistakes (Suri et al., 2024; Lampinen et al., 2024).”

    From Large Language Model Reasoning Failures (2026) · cites this paper as Lampinen et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 65

  1. [2]Newell A. 1980. Physical symbol systems. Cogn Sci. 4(2): 135– 183.
  2. [3]Marcus G. 2020. The next decade in AI: four steps towards robust artificial intelligence, arXiv, arXiv:2002.06177, preprint: not peer reviewed. 10.48550/arXiv.2002.06177 · DOI · Paper page
  3. [4]Mitchell M. 2021. Abstraction and analogy-making in artificial intelligence. Ann N Y Acad Sci. 1505(1): 79– 101. 34173249
  4. [6]Brown T, et al. 2020. Language models are few-shot learners. Adv Neural Inf Process Syst. 33: 1877– 1901.
  5. [7]Kojima T, Gu SS, Reid M, Matsuo Y, Iwasawa Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems. 35: 22199– 22213.
  6. [8]Nye M, et al. 2021a. Show your work: scratchpads for intermediate computation with language models, arXiv, arXiv:2112.00114, preprint: not peer reviewed. 10.48550/arXiv.2112.00114 · DOI · Paper page
  7. [9]Wei J, et al 2022a. Emergent abilities of large language models. Transactions on Machine Learning Research.
  8. [11]Patel A, Bhattamishra S, Goyal N. 2021. Are NLP models really able to solve simple math word problems? Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. p. 2080– 2094.
  9. [12]Razeghi Y, Logan IV RL, Gardner M, Singh S. 2022. Impact of pretraining term frequencies on few-shot reasoning. Findings of the Association for Computational Linguistics: EMNLP.
  10. [13]Valmeekam K, Olmo A, Sreedharan S, Kambhampati S. 2022. Large language models still can’t plan. NeurIPS 2022 Foundation Models for Decision Making Workshop.
  11. [14]Evans J, Barston JL, Pollard P. 1983. On the conflict between logic and belief in syllogistic reasoning. Mem Cogn. 11(3): 295– 306. 6621345
  12. [15]Kahneman D, Slovic SP, Slovic P, Tversky A. 1982. Judgment under uncertainty: heuristics and biases. Cambridge (UK): Cambridge University Press.
  13. [17]Klauer KC, Musch J, Naumer B. 2000. On belief bias in syllogistic reasoning. Psychol Rev. 107(4): 852– 884. 11089409
  14. [18]Johnson-Laird PN. 1999. Deductive reasoning. Annu Rev Psychol. 50(1): 109– 135. 15012459
  15. [19]Wason PC. 1968. Reasoning about a rule. Q J Exp Psychol. 20(3): 273– 281. 5683766
  16. [21]Cosmides L. 1989. The logic of social exchange: has natural selection shaped how humans reason? studies with the Wason selection task. Cognition. 31(3): 187– 276. 2743748
  17. [22]Cohen AL, Sidlowski S, Staub A. 2017. Beliefs and Bayesian reasoning. Psychon Bull Rev. 24(3): 972– 978. 27604495
  18. [25]Johnson-Laird PN, Legrenzi P, Legrenzi MS. 1972. Reasoning and a sense of reality. Br J Psychol. 63(3): 395– 400.
  19. [26]Marcus GF. 2003. The algebraic mind: integrating connectionism and cognitive science. Cambridge (MA): MIT Press.
  20. [27]McClelland JL, et al. 2010. Letting structure emerge: connectionist and dynamical systems approaches to cognition. Trends Cogn Sci (Regul Ed). 14(8): 348– 356. PMC3056446 20598626
  21. [28]Santoro A, Lampinen A, Mathewson K, Lillicrap T, Raposo D. 2021. Symbolic behaviour in artificial intelligence, arXiv, arXiv:2102.03406, preprint: not peer reviewed. 10.48550/arXiv.2102.03406 · DOI
  22. [29]Bommasani R, et al. 2021. On the opportunities and risks of foundation models, arXiv, arXiv:2108.07258, preprint: not peer reviewed. 10.48550/arXiv.2108.07258 · DOI · Paper page
  23. [31]Misra K, Ettinger A, Rayz JT. 2021. Do language models learn typicality judgments from text? Proceedings of the Annual Meeting of the Cognitive Science Society. 43.
  24. [33]Trinh TH, Le QV. 2018. A simple method for commonsense reasoning, arXiv, arXiv:1806.02847, preprint: not peer reviewed. 10.48550/arXiv.1806.02847 · DOI · Paper page
  25. [34]Rae JW, et al. 2021. Scaling language models: methods, analysis & insights from training gopher, arXiv, arXiv:2112.11446, preprint: not peer reviewed. 10.48550/arXiv.2112.11446 · DOI · Paper page
  26. [35]Søgaard A. 2021. Locke’s holiday: belief bias in machine reading. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. p. 8240– 8245.
  27. [36]Anil R, et al. 2023. Palm 2 technical report, arXiv, arXiv:2305.10403, preprint: not peer reviewed. 10.48550/arXiv.2305.10403 · DOI · Paper page
  28. [37]Hoffmann J, et al 2022. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems. (35): 30016– 30030.
  29. [39]MacCartney B, Manning CD. 2007. Natural logic for textual inference. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. Association for Computational Linguistics, Prague. p 193–200.
  30. [40]Bowman SR, Angeli G, Potts C, Manning CD. 2015. A large annotated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. p. 632–642.
  31. [41]Dasgupta I, Guo D, Stuhlmüller A, Gershman SJ, Goodman ND. 2018. Evaluating compositionality in sentence embeddings, arXiv, arXiv:1802.04302, preprint: not peer reviewed. 10.48550/arXiv.1802.04302 · DOI
  32. [43]Tessler MH, Tenenbaum JB, Goodman ND. 2022. Logic, probability, and pragmatics in syllogistic reasoning. Top Cogn Sci. 14(1): 574– 601. 35005842
  33. [44]Dube C, Rotello CM, Heit E. 2010. Assessing the belief bias effect with ROCs: it’s a response bias effect. Psychol Rev. 117(3): 831– 863. 20658855
  34. [45]Wason PC, Shapiro D. 1971. Natural and contrived experience in a reasoning problem. Q J Exp Psychol. 23(1): 63– 71.
  35. [46]Cox JR, Griggs RA. 1982. The effects of experience on performance in Wason’s selection task. Mem Cogn. 10(5): 496– 502. 7176911
  36. [47]Yachanin SA, Tweney RD. 1982. The effect of thematic content on cognitive strategies in the four-card selection task. Bull Psychon Soc. 19(2): 87– 90.
  37. [48]Cresswell C, Speelman CP. 2020. Does mathematics training lead to better logical thinking and reasoning? a cross-sectional assessment from students to professors. PLoS One. 15(7): 1– 21. PMC7390332 32726352
  38. [50]Evans JSB, Curtis-Holmes J. 2005. Rapid responding increases belief bias: evidence for the dual-process theory of reasoning. Think Reason. 11(4): 382– 389.
  39. [52]Wei J, et al 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems. 35: 24824– 24837.
  40. [53]Team G, et al. 2023. Gemini: a family of highly capable multimodal models, arXiv, arXiv:2312.11805, preprint: not peer reviewed. 10.48550/arXiv.2312.11805 · DOI · Paper page
  41. [54]Team G, et al. 2024. Gemma: open models based on gemini research and technology, arXiv, arXiv:2403.08295, preprint: not peer reviewed. 10.48550/arXiv.2403.08295 · DOI · Paper page
  42. [56]Goodkind A, Bicknell K. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. Proceedings of the 8th workshop on cognitive modeling and computational linguistics (CMCL 2018). Association for Computational Linguistics, Salt Lake City, Utah. p. 10–18.
  43. [58]Khashabi D, Baral C, Choi Y, Hajishirzi H. 2022. Reframing instructional prompts to GPTk’s language. In: Findings of the association for computational linguistics: ACL 2022. Association for Computational Linguistics, Dublin, Ireland. p. 589–612.
  44. [60]Mercier H, Sperber D. 2017. The enigma of reason. Cambridge (MA): Harvard University Press.
  45. [61]Evans JSB. 1984. Heuristic and analytic processes in reasoning. Br J Psychol. 75(4): 451– 468.
  46. [62]Evans JSB. 2003. In two minds: dual-process accounts of reasoning. Trends Cogn Sci (Regul Ed). 7(10): 454– 459. 14550493
  47. [63]Oaksford M, Chater N. 2003. Optimal data selection: revision, review, and reevaluation. Psychon Bull Rev. 10(2): 289– 318. 12921410
  48. [66]Botvinick MM, Cohen JD. 2014. The computational and neural basis of cognitive control: charted territory and new frontiers. Cogn Sci. 38(6): 1249– 1285. 25079472
  49. [67]Duncan J, Assem M, Shashidhara S. 2020. Integrated intelligence from distributed brain activity. Trends Cogn Sci (Regul Ed). 24(10): 838– 852. PMC7116395 32771330
  50. [68]Gershman SJ, Horvitz EJ, Tenenbaum JB. 2015. Computational rationality: a converging paradigm for intelligence in brains, minds, and machines. Science. 349(6245): 273– 278. 26185246
  51. [69]Lieder F, Griffiths TL. 2020. Resource-rational analysis: understanding human cognition as the optimal use of limited computational resources. Behav Brain Sci. 43: 1– 60. 30714890
  52. [71]Webson A, Pavlick E. 2021. Do prompt-based models really understand the meaning of their prompts? Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. p. 2300– 2344.
  53. [72]McClelland JL, Hill F, Rudolph M, Baldridge J, Schütze H. 2020. Placing language in an integrated understanding system: next steps toward human-level performance in neural language models. Proc Natl Acad Sci USA. 117(42): 25966– 25974. 32989131 PMC7585006
  54. [73]Alibali MW, Boncoddo R, Hostetter AB. 2014. Gesture in reasoning: an embodied perspective. In: The Routledge handbook of embodied cognition. Milton Park (UK): Routledge. p. 150.
  55. [74]Clark HH. 1996. Using language. Cambridge (UK): Cambridge University Press.
  56. [75]Schlangen D. 2022. Norm participation grounds language. Proceedings of the 2022 CLASP Conference on (Dis) embodiment.
  57. [76]Lehman DR, Nisbett RE. 1990. A longitudinal study of the effects of undergraduate training on reasoning. Dev Psychol. 26(6): 952– 960.
  58. [77]Luria AK. 1971. Towards the problem of the historical nature of psychological processes. Int J Psychol. 6(4): 259– 272.
  59. [78]Wu Y. 2021. LIME: Learning inductive bias for primitives of mathematical reasoning. In: International Conference on Machine Learning. PMLR. p. 11251–11262.
  60. [80]Zelikman E, Wu Y, Mu J, Goodman ND. 2022. Star: bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems. 35: 15476– 15488.
  61. [81]Chan S, et al. 2022. Data distributional properties drive emergent in-context learning in transformers. Adv Neural Inf Process Syst. 35: 18878– 18891.
  62. [82]Prystawski B, Li M, Goodman ND. 2023. Why think step-by-step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems. 36.
  63. [83]Emami A, Trischler A, Suleman K, Cheung JCK. 2020. An analysis of dataset overlap on winograd-style tasks. Proceedings of the 28th International Conference on Computational Linguistics. p. 5855– 5865.
  64. [84]Tay Y. 2023. Ul2: unifying language learning paradigms. The Eleventh International Conference on Learning Representations ICLR, Kigali, Rwanda.
  65. [85]Holtzman A, West P, Shwartz V, Choi Y, Zettlemoyer L. 2021. Surface form competition: why the highest probability answer isn’t always right. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. p. 7034– 7051.

Cited by

Sorted by citations

Authors
Lampinen, A. K., Dasgupta, I., Chan, S. C. Y., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., & Hill, F.
Affiliations
  • Google (United States) (US)
  • Google DeepMind (United Kingdom) (GB)
Corresponding author
  • Lampinen, Andrew K
  • Dasgupta, Ishita
ORCID
Venue
PNAS Nexus
Publisher
Oxford University Press (OUP)
Published
2024-06-28
Volume
3
Issue
7
Pages
pgae233-pgae233
ISSN
2752-6542 (ISSN-L)
DOI
10.1093/pnasnexus/pgae233
PMC
PMC11250216
OpenAlex
W4400681340
Cited by
69
References
9
Citations by year
2026: 24 · 2025: 29 · 2024: 11 · 2023: 5
FWCI
9.44
Type
journal article
Language
en
Open access
Licence
Locations
  • PNAS Nexus (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, content effects, abstract thinking, natural language inference, syllogism validity, Wason selection task, human reasoning, model confidence, reaction time (psychology), prior knowledge, logical inference

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review