BLiMP: the benchmark of linguistic minimal pairs for English
Open accessAbstract
We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP), 1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Lectures on Government and Binding
Chomsky, N. (1993) · Mouton eBooks · doi:10.1515/9783110884166 · Cited by 8,557 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Self-citationUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
An empirical study of smoothing techniques for language modeling
Chen, S. F., & Goodman, J. (1999) · Computer Speech & Language · doi:10.1006/csla.1999.0128 · Cited by 2,100 · see citing works
KenLM: Faster and Smaller Language Model Queries
Heafield, K. (2011) · Cited by 1,145 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
Self-citationAssessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 733 · see citing works
Adger, D. (2003) · doi:10.1093/oso/9780199243709.001.0001 · Cited by 721 · see citing works
Core Syntax: A Minimalist Approach
Adger, D. (2003) · Cited by 698 · see citing works
Bock, K., & Miller, C. (1991) · Cognitive Psychology · doi:10.1016/0010-0285(91)90003-7 · Cited by 540 · see citing works
Colorless Green Recurrent Networks Dream Hierarchically
Gulordava, K., Bojanowski, P., Grave, É., Linzen, T., & Baroni, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1108 · Cited by 475 · see citing works
The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology
Schütze, C. T. (2016) · Cited by 469 · see citing works
Chierchia, G. (2013) · doi:10.1093/acprof:oso/9780199697977.001.0001 · Cited by 456 · see citing works
Scalable Modified Kneser-Ney Language Model Estimation
Heafield, K., Pouzyrevsky, I., Clark, J. H., & Koehn, P. (2013) · Cited by 442 · see citing works
Targeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334 · see citing works
Verbal argument structure: Events and participants
Marantz, A. (2012) · Lingua · doi:10.1016/j.lingua.2012.10.012 · Cited by 320 · see citing works
What you can cram into a single vector: Probing sentence embeddings for linguistic propertiesPreprint
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01070 · Cited by 315 · see citing works
Does String-Based Neural MT Learn Source Syntax?
Shi, X., Padhi, I., & Knight, K. K. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1159 · Cited by 302 · see citing works
Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge
Lau, J. H., Clark, A., & Lappin, S. (2016) · Cognitive Science · doi:10.1111/cogs.12414 · Cited by 247 · see citing works
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y. (2024) · arXiv (Cornell University) · doi:10.57702/tcr06vve · Cited by 237 · see citing works
Definiteness and the English Existential
Ward, G., & Birner, B. J. (1995) · Language · doi:10.2307/415742 · Cited by 237 · see citing works
Uncovering the Richness of the Stimulus: Structure Dependence and Indirect Statistical Evidence
Reali, F., & Christiansen, M. H. (2005) · Cognitive Science · doi:10.1207/s15516709cog0000_28 · Cited by 209 · see citing works
What do RNN Language Models Learn about Filler–Gap Dependencies?
Wilcox, E., Lévy, R., Morita, T., & Futrell, R. (2018) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w18-5423 · Cited by 159 · see citing works
At Least et al.: The Semantics of Scalar Modifiers
Geurts, B., & Nouwen, R. (2007) · Language · doi:10.1353/lan.2007.0115 · Cited by 144 · see citing works
An Introduction to Syntactic Analysis and Theory
Koopman, H. J., Sportiche, D., & Stabler, E. P. (2013) · MPG.PuRe (Max Planck Society) · Cited by 136 · see citing works
Investigating variation in island effects
Kush, D., Lohndal, T., & Sprouse, J. (2017) · Natural Language & Linguistic Theory · doi:10.1007/s11049-017-9390-z · Cited by 106 · see citing works
Investigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs
Warstadt, A., Cao, Y. H., Grosu, I., Peng, W., Blix, H., Nie, Y., Alsop, A., Bordia, S., Liu, H., Parrish, A., et al. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1286 · Cited by 100 · see citing works
Self-citationPredicting Grammaticality on an Ordinal Scale
Heilman, M., Cahill, A., Madnani, N., Lopez, M., Mulholland, M., & Tetreault, J. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2029 · Cited by 76 · see citing works
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
Pruksachatkun, Y., Yeres, P., Liu, H., Phang, J., Htut, P. M., Wang, A., Tenney, I., & Bowman, S. R. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.15 · Cited by 67 · see citing works
Self-citationJumelet, J., & Hupkes, D. (2018) · doi:10.18653/v1/w18-5424 · Cited by 63 · see citing works
RNN Simulations of Grammaticality Judgments on Long-distance Dependencies
Chowdhury, S. A., & Zamparelli, R. (2018) · Iris (University of Trento) · Cited by 58 · see citing works
Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
Wilcox, E., Qian, P., Futrell, R., Ballesteros, M., & Lévy, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1334 · Cited by 57 · see citing works
RNNs as psycholinguistic subjects: Syntactic state and grammatical dependencyPreprint
Futrell, R., Wilcox, E., Morita, T., & Lévy, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.01329 · Cited by 50 · see citing works
Verb Argument Structure Alternations in Word and Sentence EmbeddingsPreprint
Kann, K., Warstadt, A., Williams, A., & Bowman, S. R. (2019) · arXiv (Cornell University) · doi:10.7275/q5js-4y86 · Cited by 34 · see citing works
Self-citationLinguistic Analysis of Pretrained Sentence Encoders with Acceptability JudgmentsPreprint
Warstadt, A., & Bowman, S. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.03438 · Cited by 31 · see citing works
Self-citationGrammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments.Preprint
Warstadt, A., & Bowman, S. R. (2019) · arXiv (Cornell University) · Cited by 19 · see citing works
Self-citationWhat Don't RNN Language Models Learn About Filler-Gap Dependencies?
Chaves, R. P. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/f7yj-1n62 · Cited by 18 · see citing works
Costa, J. K. D., & Chaves, R. P. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/3sb6-4g20 · Cited by 11 · see citing works
An LSTM Adaptation Study of (Un)grammaticality
Chowdhury, S. A., & Zamparelli, R. (2019) · doi:10.18653/v1/w19-4821 · Cited by 7 · see citing works
Representation of Constituents in Neural Language Models: Coordination Phrase as a Case StudyPreprint
An, A., Qian, P., Wilcox, E., & Lévy, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04625 · Cited by 3 · see citing works
Flagged first, then by citations
3 of the 51 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In Proceedings of the 27th International Conference on Computational Linguistics , pages 17901801. Association for Computational Linguistics.
- David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia , 4(1):34.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR , abs/1609.07843. · arXiv
- Ivan A. Sag, Thomas Wasow, and Emily M. Bender. 2003. Syntactic Theory: A Formal Introduction , 2 edition. CSLI Publications.
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In Proceedings of ICLR .
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A stickier benchmark for generalpurpose language understanding systems. In 33rd Conference on Neural Information Processing Systems .
Cited by
Sorted by citations
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
Evaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 265 · see citing works
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
Sinha, K., Jia, R., Hupkes, D., Pineau, J., Williams, A., & Kiela, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.230 · Cited by 204 · see citing works
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169 · see citing works
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
Jeretič, P., Warstadt, A., Bhooshan, S., & Williams, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.768 · Cited by 83 · see citing works
Self-citationFindings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
“Two weeks before the results deadline, we released three hidden evaluation tasks: the Mixed Signals Generalization Set (MSGS), a supplement to BLiMP, and an age-of-acquisition (AoA) prediction task.”
InfluentialHuang, K., Arehalli, S., Kugemoto, M., Muxica, C., Prasad, G., Dillon, B., & Linzen, T. (2023) · Journal of Memory and Language · doi:10.1016/j.jml.2024.104510 · Cited by 70 · see citing works
Evaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65 · see citing works
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.10 · Cited by 61 · see citing works
Linguistically inspired roadmap for building biologically reliable protein language models
Vu, M. H., Akbar, R., Robert, P. A., Świątczak, B., Sandve, G. K., Greiff, V., & Haug, D. T. T. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00637-1 · Cited by 57 · see citing works
Can neural networks acquire a structural bias from raw linguistic data?Preprint
Warstadt, A., & Bowman, S. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06761 · Cited by 45 · see citing works
Self-citationThe Zero Resource Speech Challenge 2021: Spoken Language Modelling
Dunbar, E., Bernard, M., Hamilakis, N., Nguyen, T. A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., & Emmanuel Dupoux, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1755 · Cited by 43 · see citing works
Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37 · see citing works
Gubelmann, R. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00696-1 · Cited by 36 · see citing works
On the Robustness of Language Encoders against Grammatical Errors
Yin, F., Long, Q., Meng, T., & Chang, K.-W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.310 · Cited by 31 · see citing works
Language acquisition: do children and language models follow similar learning stages?
Evanson, L., Lakretz, Y., & King, J. R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.773 · Cited by 27 · see citing works
Testing the limits of natural language models for predicting human language judgements
Golan, T., Siegelman, M., Kriegeskorte, N., & Baldassano, C. A. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00718-1 · Cited by 26 · see citing works
Discovering the Compositional Structure of Vector Representations with Role Learning Networks
Soulos, P., McCoy, T., Linzen, T., & Smolensky, P. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.23 · Cited by 24 · see citing works
DEMETR: Diagnosing Evaluation Metrics for Translation
Karpinska, M., Raj, N., Thai, K., Song, Y., Gupta, A., & Iyyer, M. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.649 · Cited by 22 · see citing works
SCALa: A blueprint for computational models of language acquisition in social contextPreprint
Tsuji, S., Cristià, A., & Emmanuel Dupoux, A. (2021) · doi:10.1016/j.cognition.2021.104779 · Cited by 19 · see citing works
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
Trained on 100 million words and still in shape: BERT meets British National Corpus
Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.146 · Cited by 14 · see citing works
Recurrent babbling: evaluating the acquisition of grammar from limited input data
Pannitto, L., & Herbelot, A. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.13 · Cited by 12 · see citing works
Philosophy of cognitive science in the age of deep learning
Millière, R. (2024) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1684 · Cited by 11 · see citing works
Learning Interpretable Style Embeddings via Prompting LLMs
Patel, A., Rao, D., Kothary, A., McKeown, K., & Callison-Burch, C. (2023) · arXiv · doi:10.48550/arxiv.2305.12696 · Cited by 10 · see citing works
Finding Structure in One Child's Linguistic Experience.
Wang, W., Vong, W. K., Kim, N., & Lake, B. (2023) · Cognitive Sciences · doi:10.1111/cogs.13305 · Cited by 23 · see citing works
Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9 · see citing works
Overestimation of Syntactic Representation in Neural Language Models
Kodner, J., & Gupta, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.160 · Cited by 9 · see citing works
Neural Generative Models and the Parallel Architecture of Language: A Critical Review and Outlook
Rambelli, G., Chersoni, E., Testa, D., Blache, P., & Lenci, A. (2024) · Topics in Cognitive Science · doi:10.1111/tops.12733 · Cited by 6 · see citing works
The Perceptimatic English Benchmark for Speech Perception ModelsPreprint
Millet, J., & Dunbar, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03418 · Cited by 6 · see citing works
Multilingual capabilities of GPT: A study of structural ambiguity
Yoo, M. H., Kim, J.-S., & Song, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326943 · Cited by 5 · see citing works
Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
Lakretz, Y., Desbordes, T., Hupkes, D., & Dehaene, S. (2021) · arXiv · doi:10.48550/arxiv.2110.07240 · Cited by 4 · see citing works
Boeve, S., & Bogaerts, L. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02774-4 · Cited by 4 · see citing works
Coloring the Black Box: What Synesthesia Tells Us about Character Embeddings
Kann, K., & Monsalve‐Mercado, M. M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.230 · Cited by 4 · see citing works
Rigorously assessing natural language explanations of neurons
Huang, J., Geiger, A., D’Oosterlinck, K., Wu, Z., & Potts, C. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.24 · Cited by 3 · see citing works
Meaning creation in novel noun-noun compounds: humans and language models
Chen, P., Poeppel, D., & Zuanazzi, A. (2023) · Language Cognition and Neuroscience · doi:10.1080/23273798.2023.2254865 · Cited by 3 · see citing works
Boeve, S., & Bogaerts, L. (2024) · doi:10.31219/osf.io/vqnw6 · Cited by 3 · see citing works
CLIMB – Curriculum Learning for Infant-inspired Model Building
Martinez, R. D., McGovern, H., Goriely, Z., Davis, C., Caines, A., Buttery, P., & Beinborn, L. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.10 · Cited by 2 · see citing works
Language Models Fail to Introspect About Their Knowledge of Language
Song, S., Hu, J., & Mahowald, K. (2025) · arXiv · doi:10.48550/arxiv.2503.07513 · Cited by 2 · see citing works
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C., Hu, J., Pramod, R. T., Grand, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.09605 · Cited by 2
“T2 : AGENT-2 is AGENT-1 ’s teacher.”
tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework
Sileo, D. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2h4pkr685mkq · Cited by 2 · see citing works
ToddlerBERTa: Exploiting BabyBERTa for Grammar Learning and Language Understanding
Veysel Çağatan, Ö. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.14 · Cited by 1 · see citing works
Too Much Information: Keeping Training Simple for BabyLMs
Edman, L., & Bylinina, L. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.8 · Cited by 1 · see citing works
Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
Govindarajan, V. S., Rodriguez, J. D., Bostrom, K., & Mahowald, K. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.27 · Cited by 1 · see citing works
Catasso, N. (2026) · AI-Linguistica Linguistic Studies on AI-Generated Texts and Discourses · doi:10.62408/ai-ling.v3i1.88 · Cited by 1 · see citing works
Toward context-based text-to-3D scene generation
Henlein, A. (2022) · doi:10.21248/gups.73448 · Cited by 1 · see citing works
The Learnability of the Wh-Island Constraint in Dutch by a Long Short-Term Memory NetworkPreprint
Suijkerbuijk, M., Frank, S. L., & de Swart, P. (2023) · doi:10.31234/osf.io/njr3z · Cited by 1 · see citing works
Artificial neural networks to analyze and simulate language acquisition in childrenPreprint
Lavechin, M. (2023) · doi:10.31234/osf.io/5p8ge
Boeve, S., & Bogaerts, L. (2025) · doi:10.31219/osf.io/vqnw6_v2
Omolaoye, V. A., Owoyele, B., & de Melo, G. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1198
Masked Diffusion Language Models with Frequency-Informed Training
Kosmopoulou, D., Georgiou, E., Dorovatas, V., Paraskevopoulos, G., & Potamianos, A. (2025) · BabyLM Workshop · doi:10.18653/v1/2025.babylm-main.38
Evaluating Large Language Models on Lithuanian Grammatical Cases
Jakubauskaitė, U., & Alhama, R. G. (2026) · Workshop on Language Models for Low-Resource Languages (LoResLM) · doi:10.18653/v1/2026.loreslm-1.32
Modelling non-native speech perception using automatic speech processing
Millet, J. (2022) · theses.fr (ABES) · doi:10.70675/86b4843az71c8z41d4za9dcz3db2229b1254
Zhao, J., & Brennan, J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.15.725490
Syntactic Priming in Few-Shot Learning: How Demonstration Structure Shapes LLM Performance
Yadla, P. (2026) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2026.starsem-conference.2
Gradient Starvation in Low-Rank Language Model Pretraining
Omolaoye, V. A., & de Melo, G. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.360
Causal, Hybrid, and Ternary Transformers under a Restricted Data Budget
Srivastav, A. (2026) · BabyLM Workshop · doi:10.18653/2026.babylm-main.23
- Authors
- Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R.
- Affiliations
- New York University (US)
- Corresponding author
- Warstadt, Alex
- Parrish, Alicia
- Liu, Haokun
- Mohananey, Anhad
- Peng, Wei
- Wang, Sheng-Fu
- Bowman, Samuel R.
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Issue
- 1
- Pages
- 377-392
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2990241049
- Cited by
- 28
- References
- 51
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 6 · 2022: 1 · 2021: 6 · 2020: 11
- FWCI
- 2.36
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- ScholarWorks@UMassAmherst (University of Massachusetts Amherst) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- University of Massachusetts (UMass) Amherst
- Funders
- National Science Foundation (1850208)
- Nvidia
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- grammaticality, language model evaluation, grammatical knowledge, English grammar, morphological agreement, negative polarity items, syntactic phenomena
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review