Single‐Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
The disambiguation of a syntactically ambiguous sentence in favor of a less preferred parse can lead to slower reading at the disambiguation point. This phenomenon, referred to as a garden‐path effect, has motivated models in which readers initially maintain only a subset of the possible parses of the sentence, and subsequently require time‐consuming reanalysis to reconstruct a discarded parse. A more recent proposal argues that the garden‐path effect can be reduced to surprisal arising in a fully parallel parser: words consistent with the initially dispreferred but ultimately correct parse are simply less predictable than those consistent with the incorrect parse. Since predictability has pervasive effects in reading far beyond garden‐path sentences, this account, which dispenses with reanalysis mechanisms, is more parsimonious. Crucially, it predicts a linear effect of surprisal: the garden‐path effect is expected to be proportional to the difference in word surprisal between the ultimately correct and ultimately incorrect interpretations. To test this prediction, we used recurrent neural network language models to estimate word‐by‐word surprisal for three temporarily ambiguous constructions. We then estimated the slowdown attributed to each bit of surprisal from human self‐paced reading times, and used that quantity to predict syntactic disambiguation difficulty. Surprisal successfully predicted the existence of garden‐path effects, but drastically underpredicted their magnitude, and failed to predict their relative severity across constructions. We conclude that a full explanation of syntactic disambiguation difficulty may require recovery mechanisms beyond predictability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
“Cloze Procedure”: A New Tool for Measuring Readability
Taylor, W. L. (1953) · Journalism Quarterly · doi:10.1177/107769905303000401 · Cited by 2,332
Evaluating amazon’s mechanical turk as a tool for experimental behavioral research
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057410 · Cited by 1,617
Distributed representations, simple recurrent networks, and grammatical structure
Elman, J. L. (1991) · Machine Learning · doi:10.1007/bf00114844 · Cited by 1,354
A probabilistic earley parser as a psycholinguistic model
Hale, J. (2001) · Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001 - NAACL '01 · doi:10.3115/1073336.1073357 · Cited by 1,150
Neural Network Methods for Natural Language Processing
Goldberg, Y. (2017) · Synthesis Lectures on Human Language Technologies · doi:10.1007/978-3-031-02165-7 · Cited by 821
Contextual effects on word perception and eye movements during reading
Ehrlich, S. F., & Rayner, K. (1981) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(81)90220-6 · Cited by 796
Learning Accurate, Compact, and Interpretable Tree Annotation
Petrov, S., Barrett, L., Thibaux, R., & Klein, D. (2006) · Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL - ACL '06 · doi:10.3115/1220175.1220230 · Cited by 793
Cited by
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
Oh, B.-D., & Schuler, W. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00548 · Cited by 107
Clinical efficacy of pre-trained large language models through the lens of aphasia
Cong, Y., LaCroix, A. N., & Lee, J. (2024) · Scientific Reports · doi:10.1038/s41598-024-66576-y · Cited by 25
Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9
Self-citation
Metadata
- Authors
- van Schijndel, M., & Linzen, T.
- Venue
- Cognitive Science
- Publisher
- Wiley
- Published
- 2021-06
- Volume
- 45
- Issue
- 6
- Pages
- e12988-e12988
- ISSN
- 0364-0213 (ISSN-L)
- 1551-6709
- OpenAlex
- W3166464178
- Cited by
- 68
- References
- 14
- FWCI
- 5.49
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.