A Systematic Assessment of Syntactic Generalization in Neural Language Models
Open accessAbstract
While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge.Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations.We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites.We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures.Factorially manipulating model architecture and training dataset size (1M-40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments.Our results also reveal a dissociation between perplexity and syntactic generalization performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GPT-3 is evaluated as the model with outperforming BLEU score among four state-of-art text generation methods (RNN, GAN, GPT, and CTRL) and the best syntactic generalization score among trained and off-the-shelf models.
“It is evaluated as the model with outperforming bilingual evaluation understudy (BLEU) score among four state-of-art text generation methods (RNN, GAN, GPT, and CTRL) [66] and the best syntactic generalization score among trained (LSTM, ON-LSTM, Transformer, n-gram) and ofthe-shelf (GPT, JRNN, Transformer-XL) models [46].”
From Synthetic Lies: Understanding AI-Generated Misinformation and Evaluating Algorithmic and Human Solutions (2023) · cites this paper as [46]
The cited paper’s Results (§4) identify GPT-2-XL, not GPT-3, as the best-performing model on syntactic generalization score. Its Off-the-shelf models section (§3.4) lists GPT-2 and GPT-2-XL among the models tested. The BLEU-score comparison is attached to a different citation in the citing sentence and is not assessed here. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
(2017) · Cited by 11,625 · see citing works
Building a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Loper, E., & Bird, S. (2002) · doi:10.3115/1118108.1118117 · Cited by 3,248 · see citing works
Constraints on variables in syntax
Ross, J. R. (1967) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 3,203 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
NLTK: The Natural Language ToolkitPreprint
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,893 · see citing works
Frazier, L., & Rayner, K. (1982) · Cognitive Psychology · doi:10.1016/0010-0285(82)90008-1 · Cited by 1,793 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
The cognitive basis for linguistic structures1
Bever, T. G. (2013) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199677139.003.0001 · Cited by 1,301 · see citing works
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · see citing works
Semantic Influences On Parsing: Use of Thematic Role Information in Syntactic Ambiguity Resolution
Trueswell, J. C., Tanenhaus, M. K., & Garnsey, S. M. (1994) · Journal of Memory and Language · doi:10.1006/jmla.1994.1014 · Cited by 1,175 · see citing works
The independence of syntactic processing
Ferreira, F., & Clifton, C. (1986) · Journal of Memory and Language · doi:10.1016/0749-596x(86)90006-9 · Cited by 1,151 · see citing works
A probabilistic earley parser as a psycholinguistic model
Hale, J. (2001) · Second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies 2001 - NAACL '01 · doi:10.3115/1073336.1073357 · Cited by 1,150 · see citing works
Polarity sensitivity as inherent scope relations
Ladusaw, W. A. (1980) · Cited by 1,091 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944 · see citing works
(1963) · Cited by 855 · see citing works
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 733 · see citing works
The effect of word predictability on reading time is logarithmic
Smith, N. J., & Levy, R. (2013) · Cognition · doi:10.1016/j.cognition.2013.02.013 · Cited by 725 · see citing works
Syntax: A Generative Introduction
Carnie, A. (2002) · Cited by 722 · see citing works
The Pseudo-Cleft Construction in English
Higgins, F. R. (2015) · Routledge eBooks · doi:10.4324/9781315693545 · Cited by 683 · see citing works
Improved Inference for Unlexicalized Parsing
Petrov, S., & Klein, D. (2007) · Cited by 606 · see citing works
Parsing WH-constructions: Evidence for on-line gap location
Stowe, L. A. (1986) · Language and Cognitive Processes · doi:10.1080/01690968608407062 · Cited by 601 · see citing works
(2019) · doi:10.18653/v1/n19-1419 · Cited by 601 · see citing works
Recurrent Neural Network Grammars
Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1024 · Cited by 517 · see citing works
Constituency Parsing with a Self-Attentive Encoder
Kitaev, N., & Klein, D. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1249 · Cited by 483 · see citing works
Colorless Green Recurrent Networks Dream Hierarchically
Gulordava, K., Bojanowski, P., Grave, É., Linzen, T., & Baroni, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1108 · Cited by 475 · see citing works
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385 · see citing works
Targeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334 · see citing works
How can grammars help parsers?
Crain, S., & Fodor, J. D. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511597855.004 · Cited by 310 · see citing works
What do Neural Machine Translation Models Learn about Morphology?
Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., & Glass, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1080 · Cited by 307 · see citing works
Plausibility and recovery from garden paths: An eye-tracking study.
Pickering, M. J., & Traxler, M. J. (1998) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.24.4.940 · Cited by 271 · see citing works
Processing Polarity: How the Ungrammatical Intrudes on the Grammatical
Vasishth, S., Brüssow, S., Lewis, R. L., & Drenhaus, H. (2008) · Cognitive Science · doi:10.1080/03640210802066865 · Cited by 268 · see citing works
Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge
Lau, J. H., Clark, A., & Lappin, S. (2016) · Cognitive Science · doi:10.1111/cogs.12414 · Cited by 247 · see citing works
Ordered Neurons: Integrating Tree Structures into Recurrent Neural NetworksPreprint
Shen, Y., Tan, S., Sordoni, A., & Courville, A. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09536 · Cited by 222 · see citing works
Structural Change and Reanalysis Difficulty in Language Comprehension
Sturt, P., Pickering, M. J., & Crocker, M. W. (1999) · Journal of Memory and Language · doi:10.1006/jmla.1998.2606 · Cited by 220 · see citing works
Insensitivity of the Human Sentence-Processing System to Hierarchical Structure
Frank, S. L., & Bod, R. (2011) · Psychological Science · doi:10.1177/0956797611409589 · Cited by 207 · see citing works
Predictive power of word surprisal for reading times is a linear function of language model quality
Goodkind, A., & Bicknell, K. (2018) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/w18-0102 · Cited by 190 · see citing works
Lexical Guidance in Human Parsing: Locus and Processing Characteristics
Coltheart, M. (2016) · Routledge eBooks · doi:10.4324/9781315630427-42 · Cited by 183 · see citing works
The parser doesn't ignore intransitivity, after all.
Staub, A. (2007) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.33.3.550 · Cited by 159 · see citing works
What do RNN Language Models Learn about Filler–Gap Dependencies?
Wilcox, E., Lévy, R., Morita, T., & Futrell, R. (2018) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w18-5423 · Cited by 159 · see citing works
Self-citationNeural language models as psycholinguistic subjects: Representations of syntactic state
Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., & Lévy, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1004 · Cited by 154 · see citing works
Self-citationSequential vs. Hierarchical Syntactic Models of Human Incremental Sentence Processing
Fossum, V., & Lévy, R. (2012) · Cited by 76 · see citing works
Quantity doesn’t buy quality syntax with neural language models
van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65 · see citing works
Effective Inference for Generative Neural Parsing
Stern, M., Fried, D., & Klein, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1178 · Cited by 59 · see citing works
RNN Simulations of Grammaticality Judgments on Long-distance Dependencies
Chowdhury, S. A., & Zamparelli, R. (2018) · Iris (University of Trento) · Cited by 58 · see citing works
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
Wilcox, E., Qian, P., Futrell, R., Ballesteros, M., & Lévy, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1334 · Cited by 57 · see citing works
Self-citationRNNs as psycholinguistic subjects: Syntactic state and grammatical dependencyPreprint
Futrell, R., Wilcox, E., Morita, T., & Lévy, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01329 · Cited by 50 · see citing works
Self-citationModeling garden path effects without explicit hierarchical syntax.
van Schijndel, M., & Linzen, T. (2018) · eScholarship (California Digital Library) · Cited by 46 · see citing works
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works
Hierarchical Representation in Neural Language Models: Suppression and Recovery of Expectations
Wilcox, E., Lévy, R., & Futrell, R. (2019) · doi:10.18653/v1/w19-4819 · Cited by 25 · see citing works
Self-citationWhat Don't RNN Language Models Learn About Filler-Gap Dependencies?
Chaves, R. P. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/f7yj-1n62 · Cited by 18 · see citing works
A Closer Look at the Performance of Neural Language Models on Reflexive Anaphor Licensing
Hu, J., Chen, S. Y., & Lévy, R. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/67qw-mf84 · Cited by 9 · see citing works
Self-citationCoLA: The Corpus of Linguistic Acceptability (with added annotations)
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Faculty Digital Archive (New York University Florence) · Cited by 8 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
4 of the 66 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Andrew Carnie. 2012. Syntax: A generative introduc- tion, volume 18. John Wiley & Sons.
- Anastasia Giannakidou. 2011. Negative and positive polarity items: Variation, licensing, and composi- tionality. In Semantics: An international hand- book of natural language meaning, volume 3, pages 1660-1712. Berlin: Mouton de Gruyter.
- Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Un- der the hood: Using diagnostic classifiers to in- vestigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and In- terpreting Neural Networks for NLP, pages 240- 248.
- Tanya Reinhart. 1981. Definite NP anaphora and c- command domains. Linguistic Inquiry, 12(4):605- 635.
- Ethan Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. Evaluating neural net- works as models of human online language process- ing. In Proceedings of the 42nd Meeting of the Cog- nitive Science Society (CogSci 2020). To appear.
- Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballestros, and Roger P. Levy. 2019b. Structural su- pervision improves learning of non-local grammati- cal dependencies. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3302-3312, Minneapo- lis, Minnesota.
Cited by
Sorted by citations
The neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517 · see citing works
Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581318 · Cited by 288 · see citing works
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00517 · Cited by 240 · see citing works
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language
Huebner, P. A., Sulem, E., Fisher, C. L., & Roth, D. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.49 · Cited by 71 · see citing works
Szewczyk, J., & Federmeier, K. D. (2021) · Journal of Memory and Language · doi:10.1016/j.jml.2021.104311 · Cited by 69 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
Single‐Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty
van Schijndel, M., & Linzen, T. (2021) · Cognitive Science · doi:10.1111/cogs.12988 · Cited by 74 · see citing works
Semantic Structure in Deep Learning
Pavlick, E. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 76 · see citing works
Driving and suppressing the human language network using large language models
Tuckute, G., Sathe, A., Srikant, S., Taliaferro, M., Wang, M., Schrimpf, M., Kay, K., & Fedorenko, E. G. (2024) · Nature Human Behaviour · doi:10.1038/s41562-023-01783-7 · Cited by 85 · see citing works
Using Computational Models to Test Syntactic Learnability
Wilcox, E., Futrell, R., & Lévy, R. (2022) · Linguistic Inquiry · doi:10.1162/ling_a_00491 · Cited by 64 · see citing works
Self-citationLanguage in Brains, Minds, and Machines
Tuckute, G., Kanwisher, N., & Fedorenko, E. G. (2024) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-120623-101142 · Cited by 69 · see citing works
Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2024) · Neurobiology of Language · doi:10.1162/nol_a_00137 · Cited by 63 · see citing works
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.10 · Cited by 61 · see citing works
Self-citationA fine-grained comparison of pragmatic language understanding in humans and language models
Hu, J., Floyd, S., Jouravlev, O., Fedorenko, E., & Gibson, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.230 · Cited by 78 · see citing works
Self-citationLinguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Świątczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 57 · see citing works
Inductive reasoning in humans and large language models
Han, S. J., Ransom, K., Perfors, A., & Kemp, C. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101155 · Cited by 68 · see citing works
Huang, K., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., & Linzen, T. (2023) · Journal of Memory and Language · doi:10.1016/j.jml.2024.104510 · Cited by 70 · see citing works
Hao, Y., Mendelsohn, S., Sterneck, R., Martinez, R., & Frank, R. (2020) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2020.cmcl-1.10 · Cited by 48 · see citing works
Language models align with human judgments on key grammatical constructions
Hu, J., Mahowald, K., Lupyan, G., Ivanova, A. A., & Levy, R. (2024) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2400917121 · Cited by 91 · see citing works
Self-citationPrompting is not a substitute for probability measurements in large language models
Hu, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.306 · Cited by 60 · see citing works
Self-citationFrequency Effects on Syntactic Rule Learning in Transformers
Wei, J., Garrette, D., Linzen, T., & Pavlick, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.72 · Cited by 45 · see citing works
A taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41 · see citing works
A-maze of Natural Stories: Comprehension and surprisal in the Maze task
Boyce, V., & Lévy, R. (2023) · Glossa Psycholinguistics · doi:10.5070/g6011190 · Cited by 33 · see citing works
Shen, Y., Tay, Y., Zheng, C., Bahri, D., Metzler, D., & Courville, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.559 · Cited by 29 · see citing works
How much pretraining data do language models need to learn syntax?
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.118 · Cited by 26 · see citing works
Perplexity-Based Molecule Ranking and Bias Estimation of Chemical Language Models
Moret, M., Grisoni, F., Katzberger, P., & Schneider, G. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00079 · Cited by 25 · see citing works
Yedetore, A., Linzen, T., Frank, R., & McCoy, R. T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.521 · Cited by 29 · see citing works
CLiMP: A Benchmark for Chinese Language Model Evaluation
Xiang, B., Yang, C., Li, Y., Warstadt, A., & Kann, K. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.242 · Cited by 28 · see citing works
Structural Guidance for Transformer Language Models
Qian, P., Naseem, T., Lévy, R., & Astudillo, R. F. (2021) · doi:10.18653/v1/2021.acl-long.289 · Cited by 24 · see citing works
Self-citationLampinen, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00525 · Cited by 32 · see citing works
Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction
Eisape, T., Zaslavsky, N., & Lévy, R. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.49 · Cited by 24 · see citing works
What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity
Miaschi, A., Brunato⋄, D., Dell’Orletta⋄, F., & Venturi⋄, G. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.5 · Cited by 21 · see citing works
How is BERT surprised? Layerwise detection of linguistic anomalies
Li, B., Zhu, Z., Thomas, G., Xu, Y., & Rudzicz, F. (2021) · doi:10.18653/v1/2021.acl-long.325 · Cited by 21 · see citing works
Modeling rapid language learning by distilling Bayesian priors into artificial neural networks
McCoy, R. T., & Griffiths, T. L. (2025) · Nature Communications · doi:10.1038/s41467-025-59957-y · Cited by 24 · see citing works
Effective Batching for Recurrent Neural Network Grammars
Noji, H., & Oseki, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.380 · Cited by 18 · see citing works
Language Models Use Monotonicity to Assess NPI Licensing
Jumelet, J., Denić, M., Szymanik, J., Hupkes, D., & Steinert‐Threlkeld, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.439 · Cited by 17 · see citing works
Neural reality of argument structure constructions
Li, B., Zhu, Z., Thomas, G., Rudzicz, F., & Xu, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.512 · Cited by 21 · see citing works
Schuster, S., & Linzen, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.71 · Cited by 17 · see citing works
Human-like property induction is a challenge for large language modelsPreprint
Han, S. J., Ransom, K., Perfors, A., & Kemp, C. (2022) · doi:10.31234/osf.io/6mkjy · Cited by 16 · see citing works
Decoding Prompt Syntax: Analysing its Impact on Knowledge Retrieval in Large Language Models
Linzbach, S., Tressel, T., Kallmeyer, L., Dietze, S., & Jabeen, H. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543873.3587655 · Cited by 14 · see citing works
Schrödinger's tree—On syntax and neural language models
Kulmizev, A., & Nivre, J. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796788 · Cited by 15 · see citing works
Huang, K., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., & Linzen, T. (2023) · doi:10.31234/osf.io/z38u6 · Cited by 14 · see citing works
Expectations over Unspoken Alternatives Predict Pragmatic Inferences
Hu, J. J., Lévy, R., Degen, J., & Schuster, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00579 · Cited by 20 · see citing works
Self-citationDove, G. O. (2024) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2023.0149 · Cited by 12 · see citing works
Recurrent babbling: evaluating the acquisition of grammar from limited input data
Pannitto, L., & Herbelot, A. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.13 · Cited by 12 · see citing works
Mueller, A., & Linzen, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.629 · Cited by 14 · see citing works
Predicting Human Psychometric Properties Using Computational Language Models
Laverghetta, A., Nighojkar, A., Mirzakhalov, J., & Licato, J. (2022) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-031-04572-1_12 · Cited by 11 · see citing works
The Roles of Neural Networks in Language Acquisition
Portelance, E., & Jasbi, M. (2024) · Language and Linguistics Compass · doi:10.1111/lnc3.70001 · Cited by 8 · see citing works
Szewczyk, J., & Federmeier, K. D. (2021) · doi:10.31234/osf.io/rbych · Cited by 8 · see citing works
Evaluating Neural Language Models as Cognitive Models of Language Acquisition
Martínez, H. J. V., Heuser, A., Yang, C. D., & Kodner, J. (2023) · GenBench Workshop on Generalisation (Benchmarking) in NLP (GenBench) · doi:10.18653/v1/2023.genbench-1.4 · Cited by 9 · see citing works
Sequential Machine Learning for Activity Sequence Prediction from Daily Work Report Data
Alikhani, H., Le, C., Jeong, H. D., & Damnjanovic, I. D. (2023) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-13165 · Cited by 6 · see citing works
Probing for Incremental Parse States in Autoregressive Language Models
Eisape, T., Gangireddy, V., Lévy, R., & Kim, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.203 · Cited by 6 · see citing works
Characterizing English Preposing in PP constructions
Potts, C. (2024) · Journal of Linguistics · doi:10.1017/s0022226724000227 · Cited by 6 · see citing works
Language model acceptability judgements are not always robust to context
Sinha, K., Gauthier, J., Mueller, A., Misra, K., Fuentes, K., Lévy, R., & Williams, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.333 · Cited by 7 · see citing works
Self-citationBERT-Proof Syntactic Structures: Investigating Errors in Discontinuous Constituency Parsing
Coavoux, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.288 · Cited by 7 · see citing works
Informativity enhances memory robustness against interference in sentence comprehension
Xu, W., & Futrell, R. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2024.104603 · Cited by 5 · see citing works
Representing affect information in word embeddings
Zhang, Y., Chen, W., Zhang, R., & Zhang, X. (2023) · Experiments in Linguistic Meaning · doi:10.3765/elm.2.5391 · Cited by 5 · see citing works
Shin, G., & Mun, S. (2023) · Developmental Science · doi:10.1111/desc.13405 · Cited by 5 · see citing works
(2022) · doi:10.18653/v1/2022.conll-1.8 · Cited by 5 · see citing works
Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics
Zhang, Y., Gibson, E. A. F., & Davis, F. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.1 · Cited by 5 · see citing works
Shin, G., Jung, B. K., & Mun, S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100060 · Cited by 6 · see citing works
Filtered Corpus Training (FiCT) Shows that Language Models Can Generalize from Indirect Evidence
Patil, A., Jumelet, J., Chiu, Y. Y., Lapastora, A., Shen, P. S., Wang, L., Willrich, C., & Steinert‐Threlkeld, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00720 · Cited by 8 · see citing works
Investigating Critical Period Effects in Language Acquisition through Neural Language Models
Constantinescu, I., Pimentel, T., Cotterell, R., & Warstadt, A. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00725 · Cited by 7 · see citing works
Unsupervised Dependency Graph Network
Shen, Y., Tan, S., Sordoni, A., Li, P., Zhou, J., & Courville, A. C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.327 · Cited by 5 · see citing works
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle
Shen, Y., Tan, S., Sordoni, A., Reddy, S., & Courville, A. C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.132 · Cited by 5 · see citing works
Shades of zero: Distinguishing impossibility from inconceivability
Hu, J., Sosa, F., & Ullman, T. D. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104640 · Cited by 5 · see citing works
Self-citationFlexible Generation from Fragmentary Linguistic Input
Qian, P., & Lévy, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.563 · Cited by 3 · see citing works
Self-citationNeural Networks as Cognitive Models of the Processing of Syntactic Constraints
Arehalli, S., & Linzen, T. (2024) · Open Mind · doi:10.1162/opmi_a_00137 · Cited by 4 · see citing works
Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models
Pérez-Mayos, L., García, A. T., Mille, S., & Wanner, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.333 · Cited by 4 · see citing works
Can Recurrent Neural Networks Validate Usage-Based Theories of Grammar Acquisition?
Pannitto, L., & Herbelot, A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.741321 · Cited by 4 · see citing works
Garden Path Traversal in GPT-2
Jurayj, W., Rudman, W., & Eickhoff, C. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.25 · Cited by 3 · see citing works
Active use of latent tree-structured sentence representation in humans and large language models
Liu, W., Xiang, M., & Ding, N. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02297-0 · Cited by 4 · see citing works
What Can String Probability Tell Us About Grammaticality?
Hu, J., Wilcox, E., Song, S., Mahowald, K., & Lévy, R. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.611 · Cited by 3 · see citing works
Self-citationTakeda, N., Legaspi, R., Nishimura, Y., Ikeda, K., Minamikawa, A., Plötz, T., & Chernova, S. (2023) · International Conference on Intelligent Environments (IE) · doi:10.1109/ie57519.2023.10179111 · Cited by 3 · see citing works
Sieker, J., & Zarrieß, S. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.14 · Cited by 2 · see citing works
Signal Smoothing and Syntactic Choices: A Critical Reflection on the UID Hypothesis
Juzek, T. S. (2024) · Open Mind · doi:10.1162/opmi_a_00125 · Cited by 3 · see citing works
de-Dios-Flores, I., Amboage, J. P. G., & García, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.12 · Cited by 3 · see citing works
Hints on the data for language modeling of synthetic languages with transformers
Zevallos, R., & Bel, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.699 · Cited by 4 · see citing works
Large Language Models Are Partially Primed in Pronoun Interpretation
Lam, S.-Y., Zeng, Q., Zhang, K., You, C., & Voigt, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.605 · Cited by 3 · see citing works
Meaning creation in novel noun-noun compounds: humans and language models
Chen, P., Poeppel, D., & Zuanazzi, A. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2254865 · Cited by 3 · see citing works
Transformers in the loop: Polarity in neural models of language
Bylinina, L., & Tikhonov, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.455 · Cited by 4 · see citing works
Portelance, E., Frank, M. C., & Jurafsky, D. (2024) · Cognitive Science · doi:10.1111/cogs.13448 · Cited by 3 · see citing works
Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models
Wang, Y., Hu, J., Lévy, R., & Qian, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.454 · Cited by 3 · see citing works
Self-citationXu, W., & Futrell, R. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104706 · Cited by 2 · see citing works
Kerimoğlu, C. (2026) · Dil Araştırmaları · doi:10.54316/dilarastirmalari.1879242 · Cited by 2 · see citing works
Introduction to the special section on the interaction between formal and computational linguistics
Bernard, T., & Winterstein, G. (2022) · Journal of Language Modelling · doi:10.15398/jlm.v10i1.325 · Cited by 4 · see citing works
Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation
Wu, H., & Tu, K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.482 · Cited by 2 · see citing works
Surface and Contextual Linguistic Cues in Dialog Act Classification: A Cognitive Science View
Linders, G. M., & Louwerse, M. M. (2023) · Cognitive Science · doi:10.1111/cogs.13367 · Cited by 2 · see citing works
Modelling child comprehension: A case of suffixal passive construction in Korean
Shin, G., & Mun, S. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101701 · Cited by 2 · see citing works
Zhou, Y., Tarr, M. J., & Yurovsky, D. (2025) · PLoS ONE · doi:10.1371/journal.pone.0321973 · Cited by 2 · see citing works
Kobzeva, A., Arehalli, S., Linzen, T., & Kush, D. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104663 · Cited by 2 · see citing works
MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
Jumelet, J., Weissweiler, L., Nivre, J., & Bisazza, A. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.600 · Cited by 32 · see citing works
A Targeted Assessment of the Syntactic Abilities of Transformer Models for Galician-Portuguese
García, M., & Crespo-Otero, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98305-5_5 · Cited by 2 · see citing works
Scaling Laws for BERT in Low-Resource Settings
Urbizu, G., Vicente, I. S., Saralegi, X., Agerri, R., & Soroa, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.492 · Cited by 3 · see citing works
Probing for targeted syntactic knowledge through grammatical error detection
Davis, C., Bryant, C., Caines, A., Rei, M., & Buttery, P. J. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.25 · Cited by 2 · see citing works
- Authors
- Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R.
- Affiliations
- Massachusetts Institute of Technology (US)
- Harvard University (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 1725-1744
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034510440
- Cited by
- 134
- References
- 66
- Citations by year
- 2026: 11 · 2025: 10 · 2024: 17 · 2023: 40 · 2022: 26 · 2021: 25 · 2020: 5
- FWCI
- 9.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Open Philanthropy Project
- National Institutes of Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- syntactic generalization, neural language models, syntactic knowledge, model architecture, training data size, perplexity, English syntax, sequential model, language model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
