Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.

Testing AI on language comprehension tasks reveals insensitivity to underlying meaning

Open access

Abstract

Large Language Models (LLMs) are recruited in applications that span from clinical assistance and legal support to question answering and education. Their success in specialized tasks has led to the claim that they possess human-like linguistic capabilities related to compositional understanding and reasoning. Yet, reverse-engineering is bound by Moravec's Paradox, according to which easy skills are hard. We systematically assess 7 state-of-the-art models on a novel benchmark. Models answered a series of comprehension questions, each prompted multiple times in two settings, permitting one-word or open-length replies. Each question targets a short text featuring high-frequency linguistic constructions. To establish a baseline for achieving human-like performance, we tested 400 humans on the same prompts. Based on a dataset of n = 26,680 datapoints, we discovered that LLMs perform at chance accuracy and waver considerably in their answers. Quantitatively, the tested models are outperformed by humans, and qualitatively their answers showcase distinctly non-human errors in language understanding. We interpret this evidence as suggesting that, despite their usefulness in various tasks, current AI models fall short of understanding language in a way that matches humans, and we argue that this may be due to their lack of a compositional operator for regulating grammatical and semantic information.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-024-79531-8

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 GPT-3 (text-davinci-002) and ChatGPT-3.5 performance on a grammaticality judgment task that involves low-frequency constructions. Inaccurate responses are marked in red. Accurate responses are marked in green. Full model replies and detailed linguistic analyses are available (https://osf.io/dfgmr/).
Fig. 1 GPT-3 (text-davinci-002) and ChatGPT-3.5 performance on a grammaticality judgment task that involves low-frequency constructions. Inaccurate responses are marked in red. Accurate responses are marked in green. Full model replies and detailed linguistic analyses are available (https://osf.io/dfgmr/).CC BY-NC-ND · PubMed Central
Fig. 2 (A) Mean accuracy by model and setting. (B) Mean stability by model and setting.
Fig. 2 (A) Mean accuracy by model and setting. (B) Mean stability by model and setting.CC BY-NC-ND · PubMed Central
Fig. 3 (A) Mean accuracy by responding agent and setting. (B) Mean stability by responding agent and setting.
Fig. 3 (A) Mean accuracy by responding agent and setting. (B) Mean stability by responding agent and setting.CC BY-NC-ND · PubMed Central
Fig. 4 Mapping of the types of errors presented in Fig. 1 onto levels of linguistic analysis. The observations for phonology are based on the results of63.
Fig. 4 Mapping of the types of errors presented in Fig. 1 onto levels of linguistic analysis. The observations for phonology are based on the results of63.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 73 references

Research works printed but not linked: 5

  1. Moravec, H. Mind Children (Harvard University Press, 1988).
  2. Piantadosi, S. T. & Hill, F. Meaning without reference in large language models. Proceedings of the 36th Conference on Neural Information Processing Systems (2022).
  3. Leivada, E., Murphy, E. & Marcus, G. DALL-E 2 fails to reliably capture common syntactic processes. SSHO 8, 100648 (2023).
  4. McCoy, T., Yao, S., Friedman, D., Hardy, M. & Griffiths, T. L. Embers of autoregression: Understanding Large Language Models through the problems they are trained to solve. arXiv 2309.13638v1 (2023). · arXiv
  5. Beguš, G., Dąbkowski, M. & Rhodes, R. Large linguistic models: analyzing theoretical linguistic abilities of LLMs. arXiv 2305.00948 (2023). · arXiv

Cited by

See all 54

Metadata

Authors
Dentella, V., Günther, F., Murphy, E., Marcus, G., & Leivada, E.
Affiliations
  • University of Pavia (IT)
  • Humboldt-Universität zu Berlin (DE)
  • The University of Texas Health Science Center at Houston (US)
  • New York University (US)
  • Institució Catalana de Recerca i Estudis Avançats (ES)
Corresponding author
Dentella, Vittoria
Venue
Scientific Reports
Published
2024-11-14
Volume
14
Issue
1
Pages
28083-28083
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4404349757
Cited by
54
References
73
Citations by year
2026: 28 · 2025: 25 · 2024: 1
FWCI
7.95
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Funders
  • European Commission (HORIZON2020, 945413)
  • Universitat Rovira i Virgili (945413)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
Keywords
large language models, language comprehension, compositional semantics, human-AI comparison, semantic understanding, compositional reasoning
MeSH terms
Artificial Intelligence, Female, Humans, Language, Linguistics, Semantics, Comprehension

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Testing AI on language comprehension tasks reveals insensitivity to underlying meaning. https://app.cochat.ai/papers/10.1038/s41598-024-79531-8
The paper
Dentella, V., Günther, F., Murphy, E., Marcus, G., & Leivada, E. (2024). Testing AI on language comprehension tasks reveals insensitivity to underlying meaning. Scientific Reports. https://doi.org/10.1038/s41598-024-79531-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.