CoChat

Can Language Models Handle Recursively Nested Grammatical Structures? A Case Study on Comparing Models and Humans

Open access
Lampinen, Andrew · 2024 · Computational Linguistics

The DOI matches its Crossref record; 103 of the 103 references it cites were found in a registry or an index.

Abstract

How should we compare the capabilities of language models (LMs) and humans? In this article, I draw inspiration from comparative psychology to highlight challenges in these comparisons. I focus on a case study: processing of recursively nested grammatical structures. Prior work suggests that LMs cannot process these structures as reliably as humans can. However, the humans were provided with instructions and substantial training, while the LMs were evaluated zero-shot. I therefore match the evaluation more closely. Providing large LMs with a simple prompt—with substantially less content than the human training—allows the LMs to consistently outperform the human results, even in more deeply nested conditions than were tested with humans. Furthermore, the effects of prompting are robust to the particular structures and vocabulary used in the prompt. Finally, reanalyzing the existing human data suggests that the humans may not perform above chance at the difficult structures initially. Thus, large LMs may indeed process recursively nested grammatical structures as reliably as humans, when evaluated comparably. This case study highlights how discrepancies in the evaluation methods can confound comparisons of language models and humans. I conclude by reflecting on the broader challenge of comparing human and model capabilities, and highlight an important difference between evaluating cognitive models and foundation models.

Overview

Integrity signals

Registry

Crossref · 10.1162/coli_a_00525

Cites

103 of 103 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 6

  1. Chomsky, 2014, Aspects of the Theory of Syntax
  2. Frank, 2024, Experimentology: An Open Science Approach to Experimental Psychology Methods
  3. Lakretz, 2022, Can transformers process recursive nested constructions, like humans?, 3226
  4. Marcus, 2020, GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about, Technology Review
  5. Radford, 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
  6. Wang, 2022, What language model architecture and pretraining objective works best for zero-shot generalization?, 22964

Cited by

See all 53

Metadata

Authors
Lampinen, A.
Affiliations
Google DeepMind (United Kingdom) (GB)
Corresponding author
Lampinen, Andrew
Venue
Computational Linguistics
Publisher
MIT Press
Published
2024
Volume
50
Issue
4
Pages
1441-1476
ISSN
  • 0891-2017 (ISSN-L)
  • 1530-9312
OpenAlex
W4401123261
Cited by
32
References
93
Citations by year
2026: 11 · 2025: 7 · 2024: 6
FWCI
3.54
Type
journal article
Language
en
Open access
Locations
  • Computational Linguistics (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
language models, human language processing, model evaluation, zero-shot evaluation, prompting, comparative psychology, cognitive models, foundation models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.