The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Assessing the reliability of textbook data in syntax: Adger's Core Syntax
Abstract
There has been a consistent pattern of criticism of the reliability of acceptability judgment data in syntax for at least 50 years (e.g., Hill 1961), culminating in several high-profile criticisms within the past ten years (Edelman & Christiansen 2003, Ferreira 2005, Wasow & Arnold 2005, Gibson & Fedorenko 2010, in press). The fundamental claim of these critics is that traditional acceptability judgment collection methods, which tend to be relatively informal compared to methods from experimental psychology, lead to an intolerably high number of false positive results. In this paper we empirically assess this claim by formally testing all 469 (unique, US-English) data points from a popular syntax textbook (Adger 2003) using 440 naïve participants, two judgment tasks (magnitude estimation and yes–no), and three different types of statistical analyses (standard frequentist tests, linear mixed effects models, and Bayes factor analyses). The results suggest that the maximum discrepancy between traditional methods and formal experimental methods is 2%. This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%. We discuss the implications of these results for questions about the reliability of syntactic data, as well as the practical consequences of these results for the methodological options available to syntacticians.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587
Mixed-effects modeling with crossed random effects for subjects and items
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.12.005 · Cited by 8,540
Cohen, J. (1992) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep10768783 · Cited by 5,870
Bayesian t tests for accepting and rejecting the null hypothesis
Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.2.225 · Cited by 3,688
The statistical power of abnormal-social psychological research: A review.
Cohen, J. (1962) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0045186 · Cited by 1,824
Null hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Wetzels, R., Raaijmakers, J. G. W., Jakab, E., & Wagenmakers, E.-J. (2009) · Psychonomic Bulletin & Review · doi:10.3758/pbr.16.4.752 · Cited by 143
Cited by
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 1,215
The Acquisition of Heritage Languages
Montrul, S. (2015) · doi:10.1017/cbo9781139030502 · Cited by 900
Sprouse, J., Schütze, C. T., & Almeida, D. (2013) · Lingua · doi:10.1016/j.lingua.2013.07.002 · Cited by 330
Self-citationVerbal argument structure: Events and participants
Marantz, A. (2013) · Lingua · doi:10.1016/j.lingua.2012.10.012 · Cited by 318
Dentella, V., Günther, F., & Leivada, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2309583120 · Cited by 67
Metadata
- Authors
- SPROUSE, J., & ALMEIDA, D.
- Venue
- Journal of Linguistics
- Publisher
- Cambridge University Press (CUP)
- Published
- 2012-02-08
- Volume
- 48
- Issue
- 3
- Pages
- 609-652
- ISSN
- 0022-2267 (ISSN-L)
- 1469-7742
- OpenAlex
- W2111292987
- Cited by
- 181
- References
- 7
- FWCI
- 90.59
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing the reliability of textbook data in syntax: Adger's Core Syntax. https://app.cochat.ai/papers/10.1017/s0022226712000011
- The paper
- SPROUSE, J., & ALMEIDA, D. (2012). Assessing the reliability of textbook data in syntax: Adger's Core Syntax. Journal of Linguistics. https://doi.org/10.1017/s0022226712000011
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.