The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.
Can Language Models Handle Recursively Nested Grammatical Structures? A Case Study on Comparing Models and Humans
Open accessAbstract
How should we compare the capabilities of language models (LMs) and humans? In this article, I draw inspiration from comparative psychology to highlight challenges in these comparisons. I focus on a case study: processing of recursively nested grammatical structures. Prior work suggests that LMs cannot process these structures as reliably as humans can. However, the humans were provided with instructions and substantial training, while the LMs were evaluated zero-shot. I therefore match the evaluation more closely. Providing large LMs with a simple prompt—with substantially less content than the human training—allows the LMs to consistently outperform the human results, even in more deeply nested conditions than were tested with humans. Furthermore, the effects of prompting are robust to the particular structures and vocabulary used in the prompt. Finally, reanalyzing the existing human data suggests that the humans may not perform above chance at the difficult structures initially. Thus, large LMs may indeed process recursively nested grammatical structures as reliably as humans, when evaluated comparably. This case study highlights how discrepancies in the evaluation methods can confound comparisons of language models and humans. I conclude by reflecting on the broader challenge of comparing human and model capabilities, and highlight an important difference between evaluating cognitive models and foundation models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Without a fair comparison (59–63) that evaluates humans in the same settings, we cannot make strong claims about whether the properties listed in Table 1 make LLMs qualitatively unlike humans.”
From Embers of autoregression show how large language models are shaped by the problem they are trained to solve (2024) · cites this paper as [62]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 9×
“3 The models are decoder-only Transformers (Vaswani et al., 2017), using SentencePiece (Kudo and Richardson, 2018) with 32000 tokens; they are trained to predict the next token in context on a large text corpus.”
InfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 10×
“This model is 10 × smaller than Chinchilla 70B, and 25 × smaller than GPT-3.”
InfluentialNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“However, the most broadly successful neuralnetwork language-processing systems have not incorporated explicit recursion; instead they have used recurrence, and more recently, attention (Bahdanau et al., 2014; Vaswani et al., 2017).”
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“…particularly effective at increasing base model performance (Webson and Pavlick 2021) and should probably be replaced with examples if possible—though models trained or tuned to follow instructions (e.g., Raffel et al. 2020; Ouyang et al. 2022; Chung et al. 2024) would perhaps show larger bene-fits.”
Chomsky, N. (2014) · The MIT Press eBooks · doi:10.7551/mitpress/9780262527347.001.0001 · Cited by 8,244
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Chomsky, N. (2001) · Ken Hale · doi:10.7551/mitpress/4056.003.0004 · Cited by 5,105 · cited 4×
“Merge is generally described as a classical symbolic operation over structures—one that is “indispensable” (Chomsky 1999).”
Influential
Research works printed but not linked: 6
- Chomsky, 2014, Aspects of the Theory of Syntax
- Frank, 2024, Experimentology: An Open Science Approach to Experimental Psychology Methods
- Lakretz, 2022, Can transformers process recursive nested constructions, like humans?, 3226
- Marcus, 2020, GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about, Technology Review
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- Wang, 2022, What language model architecture and pretraining objective works best for zero-shot generalization?, 22964
Cited by
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
“Life is what we make of it, and it’s up to each of us to find our own meaning in life.”
Niu, Q., Liu, J., Bi, Z., Feng, P., Peng, B., Chen, K., Li, M., Yan, L. K. Q., Zhang, Y., Yin, C. H., et al. (2026) · BIO Integration · doi:10.15212/bioi-2025-0199 · Cited by 121
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322420121 · Cited by 96
(Ir)rationality and cognitive biases in large language models
Macmillan-Scott, O., & Musolesi, M. (2024) · Royal Society Open Science · doi:10.1098/rsos.240255 · Cited by 118
“Lampinen[15] discusses this problem when it comes to language models in particular, highlighting that different approaches must be taking to evaluate cognitive and foundation models.”
Background
Metadata
- Authors
- Lampinen, A.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Corresponding author
- Lampinen, Andrew
- ORCID
- Lampinen, Andrew
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 4
- Pages
- 1441-1476
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4401123261
- Cited by
- 32
- References
- 93
- Citations by year
- 2026: 11 · 2025: 7 · 2024: 6
- FWCI
- 3.54
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- language models, human language processing, model evaluation, zero-shot evaluation, prompting, comparative psychology, cognitive models, foundation models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Can Language Models Handle Recursively Nested Grammatical Structures? A Case Study on Comparing Models and Humans. https://app.cochat.ai/papers/10.1162/coli_a_00525
- The paper
- Lampinen, A. (2024). Can Language Models Handle Recursively Nested Grammatical Structures? A Case Study on Comparing Models and Humans. Computational Linguistics. https://doi.org/10.1162/coli_a_00525
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.