Can Language Models Handle Recursively Nested Grammatical Structures? A Case Study on Comparing Models and Humans
Open accessThe DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.
Abstract
How should we compare the capabilities of language models (LMs) and humans? In this article, I draw inspiration from comparative psychology to highlight challenges in these comparisons. I focus on a case study: processing of recursively nested grammatical structures. Prior work suggests that LMs cannot process these structures as reliably as humans can. However, the humans were provided with instructions and substantial training, while the LMs were evaluated zero-shot. I therefore match the evaluation more closely. Providing large LMs with a simple prompt—with substantially less content than the human training—allows the LMs to consistently outperform the human results, even in more deeply nested conditions than were tested with humans. Furthermore, the effects of prompting are robust to the particular structures and vocabulary used in the prompt. Finally, reanalyzing the existing human data suggests that the humans may not perform above chance at the difficult structures initially. Thus, large LMs may indeed process recursively nested grammatical structures as reliably as humans, when evaluated comparably. This case study highlights how discrepancies in the evaluation methods can confound comparisons of language models and humans. I conclude by reflecting on the broader challenge of comparing human and model capabilities, and highlight an important difference between evaluating cognitive models and foundation models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Without a fair comparison (59–63) that evaluates humans in the same settings, we cannot make strong claims about whether the properties listed in Table 1 make LLMs qualitatively unlike humans.”
From Embers of autoregression show how large language models are shaped by the problem they are trained to solve (2024) · cites this paper as [62]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 9×
“3 The models are decoder-only Transformers (Vaswani et al., 2017), using SentencePiece (Kudo and Richardson, 2018) with 32000 tokens; they are trained to predict the next token in context on a large text corpus.”
InfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 10×
“This model is 10 × smaller than Chinchilla 70B, and 25 × smaller than GPT-3.”
InfluentialNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“However, the most broadly successful neuralnetwork language-processing systems have not incorporated explicit recursion; instead they have used recurrence, and more recently, attention (Bahdanau et al., 2014; Vaswani et al., 2017).”
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“…particularly effective at increasing base model performance (Webson and Pavlick 2021) and should probably be replaced with examples if possible—though models trained or tuned to follow instructions (e.g., Raffel et al. 2020; Ouyang et al. 2022; Chung et al. 2024) would perhaps show larger bene-fits.”
Chomsky, N. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9780262527347.001.0001 · Cited by 8,244
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Chomsky, N. (2001) · Ken Hale · doi:10.7551/mitpress/4056.003.0004 · Cited by 5,105 · cited 4×
“Merge is generally described as a classical symbolic operation over structures—one that is “indispensable” (Chomsky 1999).”
Influential
Research works printed but not linked: 6
- Chomsky, 2014, Aspects of the Theory of Syntax
- Frank, 2024, Experimentology: An Open Science Approach to Experimental Psychology Methods
- Lakretz, 2022, Can transformers process recursive nested constructions, like humans?, 3226
- Marcus, 2020, GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about, Technology Review
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Wang, 2022, What language model architecture and pretraining objective works best for zero-shot generalization?, 22964
Cited by
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
“Life is what we make of it, and it’s up to each of us to find our own meaning in life.”
Niu, Q., Liu, J., Bi, Z., Feng, P., Peng, B., Chen, K., Li, M., Yan, L. K. Q., Zhang, Y., Yin, C. H., et al. (2026) · BIO Integration · doi:10.15212/bioi-2025-0199 · Cited by 121
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322420121 · Cited by 96
(Ir)rationality and cognitive biases in large language models
Macmillan-Scott, O., & Musolesi, M. (2024) · Royal Society Open Science · doi:10.1098/rsos.240255 · Cited by 47
“Lampinen[15] discusses this problem when it comes to language models in particular, highlighting that different approaches must be taking to evaluate cognitive and foundation models.”
BackgroundEvent Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely
Kauf, C., Ivanova, A. A., Rambelli, G., Chersoni, E., She, J. S., Chowdhury, Z., Fedorenko, E., & Lenci, A. (2023) · Cognitive Science · doi:10.1111/cogs.13386 · Cited by 53
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68
Self-citationEntity Tracking in Language Models
Kim, N., & Schuster, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.213 · Cited by 159
Prompting is not a substitute for probability measurements in large language models
Hu, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.306 · Cited by 60
How to evaluate the cognitive abilities of LLMs
Ivanova, A. A. (2025) · Nature Human Behaviour · doi:10.1038/s41562-024-02096-z · Cited by 46
Large languages, impossible languages and human brains
Moro, A., Greco, M., & Cappa, S. F. (2023) · Cortex · doi:10.1016/j.cortex.2023.07.003 · Cited by 61
On Benchmarking Human-like Intelligence in Machines
Ying, L., Collins, K. M., Wong, L., Sucholutsky, I., Liu, R., Weller, A., Shu, T., Griffiths, T. L., & Tenenbaum, J. B. (2026) · Open Mind · doi:10.1162/opmi.a.385 · Cited by 41
“…or sometimes loosely adapted from psychological assays are used to directly evaluate computational model performance, often with ground-truth notions of what it means to “solve” a task (for instance, to identify whether a model can label mental states in simple “false-belief” tasks derived from…”
Leveraging Large Language Models for Enhancing Safety in Maritime Operations
Miller, T. I., Durlik, I., Kostecka, E., Łobodzińska, A., Łazuga, K., & Kozlovska, P. (2025) · Applied Sciences · doi:10.3390/app15031666 · Cited by 45
LLMs as models for analogical reasoning
Musker, S., Duchnowski, A., Millière, R., & Pavlick, E. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104676 · Cited by 46
“However, LLM performance can also be negatively affected by strong auxiliary task demands (Hu & Frank, 2024) and mis-matched conditions in comparisons with human subjects (Lampinen, 2023).”
Misra, K., Rayz, J., & Ettinger, A. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 15
Kauf, C., Chersoni, E., Lenci, A., Fedorenko, E. G., & Ivanova, A. A. (2024) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2024.blackboxnlp-1.18 · Cited by 51
A roadmap for evaluating moral competence in large language models
Haas, J., Bridgers, S., Manzini, A., Henke, B., May, J., Levine, S., Weidinger, L., Shanahan, M., Lum, K., Gabriel, I., et al. (2026) · Nature · doi:10.1038/s41586-025-10021-1 · Cited by 25
Levels of analysis for large language models
Ku, A. Y., Campbell, D., Bai, X., Geng, J., Liu, R., Marjieh, R., McCoy, R. T., Nam, A., Sucholutsky, I., Zhang, L., et al. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2025.0012 · Cited by 28
Cox, S. R., Abdul, A., & Ooi, W. T. (2023) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3623809.3623931 · Cited by 32
“Similarly, recent studies have compared behaviour and task proficiency between humans and LLMs [18, 27], with some finding LLMs exceed crowd-workers in a number of tasks [12, 13, 26](such as data labelling [12, 26]), while others have found humans more proficient in nuanced creative tasks [15, 24].”
Meta-in-context learning in large language models
Coda-Forno, J., Binz, M., Akata, Z., Botvinick, M., Wang, J., & Schulz, E. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2844 · Cited by 20
Philosophy of cognitive science in the age of deep learning
Millière, R. (2024) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1684 · Cited by 27
Dentella, V., Günther, F., & Leivada, E. (2025) · PLoS ONE · doi:10.1371/journal.pone.0327794 · Cited by 22
“While it is true that attempting several prompting styles can aid the LLMs in providing correct answers, it is not clear that such practice would contribute to establishing a fair comparison with humans (cf. Lampinen, 2023 for considerations on the comparability of human/LLM evaluation methods).”
BackgroundNiu, Q., Liu, J., Bi, Z., Feng, P., Peng, B., Chen, K., Li, M., Yan, L. K., Zhang, Y., Yin, C. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.02387 · Cited by 16
Large Language Models Estimate Fine‐Grained Human Color–Concept Associations
Mukherjee, K., Mohapatra, A., Rogers, T. T., & Schloss, K. B. (2026) · Cognitive Science · doi:10.1111/cogs.70219 · Cited by 15
“…et al., 2020) families—also produce human-like behaviors in psychological tasks involving both language (Futrell et al., 2019; Kumar et al., 2022; Lampinen, 2022; Piantadosi & Hill, 2022; Tuckute et al., 2023) and perception (Conwell, Prince, Alvarez, & Konkle, 2023; Mirchandani et al., 2023).”
Whither symbols in the era of advanced neural networks?
Griffiths, T. L., Lake, B. M., McCoy, R. T., Pavlick, E., & Webb, T. W. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2026.02.003 · Cited by 21
Language model acceptability judgements are not always robust to context
Sinha, K., Gauthier, J., Mueller, A., Misra, K., Fuentes, K., Lévy, R., & Williams, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.333 · Cited by 33
(Ir)rationality in AI: state of the art, research challenges and open questions
Macmillan-Scott, O., & Musolesi, M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11341-4 · Cited by 16
“The understanding of rationality within AI has been influenced by some of these, particularly by work within economics, philosophy, and psychology.”
Overinformative Question Answering by Humans and MachinesPreprint
Tsvilodub, P., Franke, M., Hawkins, R. D., & Goodman, N. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.07151 · Cited by 11
“…the mechanisms of these models have revealed intriguing differences between LLM performance and humans on cognitive tasks commonly used in psychology (Binz & Schulz, 2022), as well as the sensitivity of these models to the structure of their prompts (Lampinen, 2022; Lampinen et al., 2022).”
BackgroundRe-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10
Allaway, E., Bhagavatula, C., Hwang, J. D., McKeown, K. R., & Leslie, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00530 · Cited by 25
What Can String Probability Tell Us About Grammaticality?
Hu, J., Wilcox, E., Song, S., Mahowald, K., & Lévy, R. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.611 · Cited by 17
Luo, X., Ramscar, M., & Love, B. C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.11061 · Cited by 9
“For example, models are frequently tested in zero-shot settings while humans receive task-specific training and instructions 7 .”
Anthropocentric Bias in Language Model Evaluation
Millière, R., & Rathkopf, C. (2026) · Computational Linguistics · doi:10.1162/coli.a.582 · Cited by 6
“This concern is highlighted by Lampinen (2023)’s case study comparing human and model performance on recursively nested grammatical structures, in response to prior work by Lakretz et al. (2022).”
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.38 · Cited by 39
Harnessing Language Models for Studying the Ancient Greek Language: A Systematic Review
Tzanoulinou, D., Triantafyllopoulos, L., & Verykios, V. S. (2025) · Machine Learning and Knowledge Extraction · doi:10.3390/make7030071 · Cited by 5
Misra, K., Ettinger, A., & Mahowald, K. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.687 · Cited by 5
de-Dios-Flores, I., Amboage, J. P. G., & García, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.12 · Cited by 4
“Lampinen (2022), however, questions these comparisons between humans and LMs, as the former receive guidance before the experiments, while LMs are evaluated on zero-shot scenarios, and their performance improves with few-shot prompts.”
BackgroundThe compositionality continuum as a principle for studying the neural basis of intelligence
Riveland, R., Pouget, A., & Driscoll, L. N. (2026) · Nature Neuroscience · doi:10.1038/s41593-026-02382-1 · Cited by 6
“Providing practice demonstrations to LLMs resulted in performance comparable to that of humans 83 .”
Apidianaki, M., Fourtassi, A., & Padó, S. (2024) · Computational Linguistics · doi:10.1162/coli_e_00539 · Cited by 5
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.09605 · Cited by 2
“Drawing inspiration from comparative psychology, such an approach of matched evaluation has been proposed as a way to support increasingly fair evaluations of LLMs, allowing for more direct comparison of performance, consistency, and context-sensitivity with human participants (Lampinen, 2024).”
How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?
Liu, R., Sumers, T. R., Dasgupta, I., & Griffiths, T. L. (2024) · arXiv · doi:10.48550/arxiv.2402.07282 · Cited by 1
Topic-Conversation Relevance (TCR) Dataset and Benchmarks
Fan, Y., Pool, J., Filipi, S., & Cutler, R. (2024) · neural information processing systems · doi:10.52202/079017-4448 · Cited by 2
A Suite of LMs Comprehend Puzzle Statements as Well or Better Than Humans
Rakshit, S., Hu, J., Mahowald, K., & Goldberg, A. E. (2026) · Open Mind · doi:10.1162/opmi.a.344 · Cited by 3
“Decisions that can influence results include how questions are displayed, whether instructions or examples are provided (e.g., Weissweiler et al., 2025), and whether binary, ordinal, or gradient responses are collected (Hu & Frank, 2024; Ivanova, 2025; Lampinen, 2024).”
Hwang, H. (2026) · Acta Psychologica · doi:10.1016/j.actpsy.2026.107025 · Cited by 2
Neural language models and human linguistic knowledge
Hu, J. (2023) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2
Of Models and Men: Probing Neural Networks for Agreement Attraction with Psycholinguistic Data
Bazhukov, M., Волошина, Е. В., Pletenev, S., Anisimov, A., Serikov, O., & Toldova, S. (2024) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2024.conll-1.22 · Cited by 7
Reasoning with language and knowledge graphs
Vázquez, S. G. (2024) · Repositori digital de la UPF (Universitat Pompeu Fabra)
Advancing End-User Development in Robotics with Large Language Models and Digital Twins
Gargioni, L. (2025) · BOA (University of Milano-Bicocca)
Experimental versus In-Corpus Variation in Referring Expression Choice
Ellison, T. M., & Same, F. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5p6sobxbpquz
Human-Robot Collaboration in Healthcare: New Programming and Interaction Techniques
Gargioni, L. (2026) · Institutional Research Information System (Università degli Studi di Brescia)
Fang, S., Li, Y., & Cong, Y. (2026) · Languages · doi:10.3390/languages11060126
“…or semantic representations, prior work has shown that GPT-2 can capture a wide range of syntactic and semantic regularities (e.g., Hu et al., 2024; Lampinen, 2024; Li et al., 2025; Wang et al., 2025), including phenomena related to scope ambiguity (Fang et al., 2026b; Kamath et al., 2024).”
Large Language Models as Distributional Baselines for Language Tasks
Trott, S., Michaelov, J. A., Jones, C., Chang, T. A., & Bergen, B. K. (2026) · Open Mind · doi:10.1162/opmi.a.373
“If an LLM ’ s training data are open, researchers can conduct an audit of those data to search for examples of the stimuli they wish to use (see, e.g., Lampinen, 2024).”
A digital twin-enabled environment for end-user development of robot tasks
Gargioni, L., Fogli, D., & Nava, E. M. (2026) · Intelligent Service Robotics · doi:10.1007/s11370-026-00758-7
Metadata
- Authors
- Lampinen, A.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Corresponding author
- Lampinen, Andrew
- ORCID
- Lampinen, Andrew
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 4
- Pages
- 1441-1476
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4401123261
- Cited by
- 32
- References
- 93
- Citations by year
- 2026: 11 · 2025: 7 · 2024: 6
- FWCI
- 3.54
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- language models, human language processing, model evaluation, zero-shot evaluation, prompting, comparative psychology, cognitive models, foundation models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.