The DOI matches its Crossref record; 44 of the 46 references it cites were found in a registry or an index.
Targeted Syntactic Evaluation of Language Models
Open accessAbstract
We present a dataset for evaluating the grammaticality of the predictions of a language model.We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence.The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items.We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one.In an experiment using this data set, an LSTM language model performed poorly on many of the constructions.Multi-task training with a syntactic objective (CCG supertagging) improved the LSTM's accuracy, but a large gap remained between its performance and the accuracy of human participants recruited online.This suggests that there is considerable room for improvement over LSTMs in capturing syntax in a language model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“BLiMP, and other related syntactic benchmarks such as those presented in Marvin and Linzen (2018) and Gauthier et al. (2020), were specifically designed to mimic the types of tests invented by linguists and cognitive scientists to reveal syntactic competence-i.e., they are all based on minimal pair sentences.”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Marvin and Linzen (2018)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390
SRILM - an extensible language modeling toolkit
Stolcke, A. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-303 · Cited by 4,385
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 1,215
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 820
Self-citationAgreement attraction in comprehension: Representations and processes
WAGERS, M. W., Lau, E. F., & Phillips, C. (2009) · Journal of Memory and Language · doi:10.1016/j.jml.2009.04.002 · Cited by 639
Bock, K., & Miller, C. (1991) · Cognitive Psychology · doi:10.1016/0010-0285(91)90003-7 · Cited by 558
Colorless Green Recurrent Networks Dream Hierarchically
Gulordava, K., Bojanowski, P., Grave, É., Linzen, T., & Baroni, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1108 · Cited by 538
Self-citation
Research works printed but not linked: 2
- Anastasia Giannakidou. 2011. Negative and positive polarity items: Variation, licensing, and composi- tionality. In Semantics: An international hand- book of natural language meaning, pages 1660- 1712. Berlin: Mouton de Gruyter.
- Bingley, U.K.: Emerald.
Cited by
Sorted by citations
Items 201–300 of 332
Emergence of Syntax Needs Minimal Supervision
Bailly, R., & Gábor, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.46 · Cited by 4
Martínez, H. V. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.38 · Cited by 4
What Can a Generative Language Model Answer About a Passage?
Summers-Stay, D. A., Bonial, C., & Voss, C. R. (2021) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/2021.mrqa-1.7 · Cited by 6
Shin, U., Yi, E., & Song, S. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.937656 · Cited by 4
Characterizing verbatim short-term memory in neural language models
Armeni, K., Honey, C., & Linzen, T. (2022) · Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2022.conll-1.28 · Cited by 4
Self-citationNeural Networks as Cognitive Models of the Processing of Syntactic Constraints
Arehalli, S., & Linzen, T. (2024) · Open Mind · doi:10.1162/opmi_a_00137 · Cited by 6
Self-citationUncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning
Davis, F., & van Schijndel, M. (2021) · doi:10.18653/v1/2021.acl-long.93 · Cited by 7
Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models
Pérez-Mayos, L., García, A. T., Mille, S., & Wanner, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.333 · Cited by 5
Can Recurrent Neural Networks Validate Usage-Based Theories of Grammar Acquisition?
Pannitto, L., & Herbelot, A. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.741321 · Cited by 4
Unsupervised Recurrent Neural Network GrammarsPreprint
Kim, Y., Rush, A. M., Yu, L., Kuncoro, A., Dyer, C., & Melis, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03746 · Cited by 20
Detecting and Exorcising Statistical Demons from Language Models with Anti-Models of Negative DataPreprint
Wick, M., Silverstein, K., Tristan, J.-B., Pocock, A., & Johnson, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11855 · Cited by 4
Influence Patterns for Explaining Information Flow in BERTPreprint
Lu, K., Wang, Z., Mardziel, P., & Datta, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.00740 · Cited by 4
Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans
Lakretz, Y., Desbordes, T., Hupkes, D., & Dehaene, S. (2021) · arXiv · doi:10.48550/arxiv.2110.07240 · Cited by 4
Improving neural machine translation of languages with little data and rich morphology
Dhar, P. (2024) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.33612/diss.859674171 · Cited by 3
Active use of latent tree-structured sentence representation in humans and large language models
Liu, W., Xiang, M., & Ding, N. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02297-0 · Cited by 8
What Can String Probability Tell Us About Grammaticality?
Hu, J., Wilcox, E., Song, S., Mahowald, K., & Lévy, R. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.611 · Cited by 7
Sieker, J., & Zarrieß, S. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.14 · Cited by 2
Unsupervised Representation Disentanglement of Text: An Evaluation on Synthetic Datasets
Zhang, L., Prokhorov, V., & Shareghi, E. (2021) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2021.repl4nlp-1.14 · Cited by 3
Probe-Less Probing of BERT’s Layer-Wise Linguistic Knowledge with Masked Word Prediction
Aoyama, T., & Schneider, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2022.naacl-srw.25 · Cited by 3
Can Language Models Be Specific? How?
Huang, J., Chang, K. C., Xiong, J., & Hwu, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.45 · Cited by 4
de-Dios-Flores, I., Amboage, J. P. G., & García, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.12 · Cited by 4
Li, Y., Yao, K., Qin, L., Peng, S., Liu, Y., & Li, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6343 · Cited by 3
Pislar, M., & Rei, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.335 · Cited by 3
Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models
Derby, S., Miller, P. C., & Devereux, B. J. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.36 · Cited by 3
Prokhorov, V., Li, Y., Shareghi, E., & Collier, N. (2021) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2021.repl4nlp-1.5 · Cited by 6
Transformers in the loop: Polarity in neural models of language
Bylinina, L., & Tikhonov, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.455 · Cited by 4
Exploring Linguistic Acceptability in Swedish Learners’ Language
Klezl, J., Mohammed, Y. A., & Volodina, E. (2022) · Linköping electronic conference proceedings · doi:10.3384/ecp190009 · Cited by 3
SLOG: A Structural Generalization Benchmark for Semantic Parsing
Li, B., Donatelli, L. E., Koller, A., Linzen, T., Yao, Y., & Kim, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.194 · Cited by 13
Self-citationAre BERTs Sensitive to Native Interference in L2 Production?
Tang, Z., Mitra, P., & Reitter, D. (2021) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2021.insights-1.6 · Cited by 3
Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models
Wang, Y., Hu, J., Lévy, R., & Qian, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.454 · Cited by 3
Scalable Syntax-Aware Language Models Using Knowledge DistillationPreprint
Kuncoro, A., Dyer, C., Rimell, L., Clark, S. C., & Blunsom, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06438 · Cited by 7
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIsPreprint
Warstadt, A., Cao, Y. H., Grosu, I., Peng, W., Blix, H., Nie, Y., Alsop, A., Bordia, S., Liu, H., Parrish, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02597 · Cited by 8
Representation of Constituents in Neural Language Models: Coordination Phrase as a Case StudyPreprint
An, A., Qian, P., Wilcox, E., & Lévy, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04625 · Cited by 5
On the Robustness of Language Encoders against Grammatical ErrorsPreprint
Yin, F., Long, Q., Meng, T., & Chang, K.-W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05683 · Cited by 3
A unifying computational account of temporal context effects in language across the human cortexPreprint
Vo, V. A., Jain, S., Beckage, N., Chien, H.-Y. S., Obinwa, C., & Huth, A. G. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.08.03.551886 · Cited by 5
Kerimoğlu, C. (2026) · Dil Araştırmaları · doi:10.54316/dilarastirmalari.1879242 · Cited by 2
Using Roark-Hollingshead Distance to Probe BERT’s Syntactic Competence
Niu, J., Lu, W., Corlett, E., & Penn, G. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.27 · Cited by 3
Modelling child comprehension: A case of suffixal passive construction in Korean
Shin, G., & Mun, S. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101701 · Cited by 2
Toward Immersive Computational Storytelling: Card-Framework for Enhanced Persona-Driven Dialogues
Liao, B., & Vargas, D. V. (2024) · IEEE Transactions on Games · doi:10.1109/tg.2024.3466898 · Cited by 2
Investigating Idiomaticity in Word Representations
He, W. P., Vieira, T. K., García, M., Scarton, C., Idiart, M., & Villavicencio, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00546 · Cited by 4
MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
Jumelet, J., Weissweiler, L., Nivre, J., & Bisazza, A. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.600 · Cited by 66
Exploring Multilingual Syntactic Sentence Representations
Liu, C., de Andrade, A., & Osama, M. (2019) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/d19-5521 · Cited by 2
How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?
Bhatt, G., Bansal, H., Singh, R., & Agarwal, S. (2020) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2020.acl-srw.33 · Cited by 5
A Targeted Assessment of the Syntactic Abilities of Transformer Models for Galician-Portuguese
García, M., & Crespo-Otero, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98305-5_5 · Cited by 3
Assessing linguistic generalisation in language models: a dataset for Brazilian Portuguese
Wilkens, R., Zilio, L., & Villavicencio, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09664-1 · Cited by 2
Post-hoc analysis of Arabic transformer models
Abdelalí, A., Durrani, N., Dalvi, F., & Sajjad, H. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.8 · Cited by 8
Probing for targeted syntactic knowledge through grammatical error detection
Davis, C., Bryant, C., Caines, A., Rei, M., & Buttery, P. J. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.25 · Cited by 2
On General Language Understanding
Schlangen, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.591 · Cited by 3
Vietnamese Span-based Constituency Parsing with BERT Embedding
Phan, U., HUYNH, N.-T.-T., Truong, T. H., Dao, A., & Đinh, Đ. (2019) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse.2019.8919467 · Cited by 2
Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models
Derby, S., Devereux, B. J., & Miller, P. C. (2021) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2021.cmcl-1.25 · Cited by 5
Do pretrained transformers infer telicity like humans?
Zhao, Y., Ngui, J. G., Hartley, L. H., & Bethard, S. J. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.6 · Cited by 3
A Neural Model for Compositional Word Embeddings and Sentence Processing
Lappin, S., & Bernardy, J.-P. (2022) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2022.cmcl-1.2 · Cited by 2
Attribution Analysis of Grammatical Dependencies in LSTMsPreprint
Hao, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00062 · Cited by 2
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic OraclePreprint
Shen, Y., Tan, S., Sordoni, A., Reddy, S., & Courville, A. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07960 · Cited by 2
Neural approaches to discourse coherence: modeling, evaluation and application
Farag, Y. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.70024 · Cited by 2
Towards Immersive Computational Storytelling:Card-Framework for Enhanced Persona-Driven DialoguesPreprint
Liao, B., & Vargas, D. V. (2023) · doi:10.36227/techrxiv.24605667.v1 · Cited by 2
Kementchedjhieva, Y., Anderson, M., & Søgaard, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.429 · Cited by 10
Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages
Patil, R., Dhillon, J., Mahurkar, S., Kulkarni, S., Malhotra, M., & Baths, V. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.14 · Cited by 1
Wisniewski, G., Zhu, L., Ballier, N., & Yvon, F. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.13 · Cited by 1
NLMs: Augmenting Negation in Language Models
Singh, R., Kumar, R., & Sridhar, V. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.873 · Cited by 1
Training and evaluation of vector models for Galician
García, M. (2024) · Language Resources and Evaluation · doi:10.1007/s10579-024-09740-0 · Cited by 1
CoLAGaze: A Corpus of Eye Movements for Linguistic Acceptability
Bondar, A., Reich, D., & Jäger, L. (2025) · Eye Tracking Research & Application · doi:10.1145/3715669.3723120 · Cited by 5
Yoo, M. H., & Song, S. (2025) · Lingua · doi:10.1016/j.lingua.2025.103998 · Cited by 1
How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology?
Amrhein, C., & Sennrich, R. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-207983 · Cited by 1
Anderson, M., & Camacho-Collados, J. (2022) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2022.starsem-1.15 · Cited by 1
Felhi, G., Roux, J. L., & Seddah, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.423 · Cited by 3
Exploring How Generative Adversarial Networks Learn Phonological Representations
Chen, J., & Elsner, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.175 · Cited by 1
Evaluating the machine learning models based on natural language processing tasks
Meeradevi, M., Sowmya, B. J., & Swetha, B. N. (2024) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v13.i2.pp1954-1968 · Cited by 1
Raising the Bar on Acceptability Judgments Classification: An Experiment on ItaCoLA Using ELECTRA
Guarasci, R., Minutolo, A., Buonaiuto, G., Pietro, G. D., & Esposito, M. (2024) · Electronics · doi:10.3390/electronics13132500 · Cited by 1
Applying Variational Quantum Classifier on Acceptability Judgements: A QNLP Experiment
Guarasci, R., Buonaiuto, G., Pietro, G. D., & Esposito, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-81247-7_8 · Cited by 1
Evaluating the Chinese Syntactic Ability of ModernBERT
Yu, H., Sun, H., Sun, J., Shan, Q., & Tang, M. (2025) · International Annual Conference on Complex Systems and Intelligent Science (CSIS-IAC) · doi:10.1109/csis-iac65538.2025.11161212 · Cited by 1
Changing the Basis of Contextual Representations with Explicit Semantics
Ficsor, T., & Berend, G. (2021) · doi:10.18653/v1/2021.acl-srw.25 · Cited by 2
Incremental Processing of Principle B: Mismatches Between Neural Models and Humans
Davis, F. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.11 · Cited by 1
NxPlain: A Web-based Tool for Discovery of Latent Concepts
Dalvi, F., Durrani, N., Sajjad, H., Jaban, T., Husaini, M., & Abbas, U. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-demo.10 · Cited by 1
Acquiring Linguistic Knowledge from Multimodal Input
Amariucai, T., & Warstadt, A. S. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.11 · Cited by 1
Mi, M. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.23 · Cited by 1
Evaluating visually grounded language capabilities using microworlds
Kuhnle, A. (2019) · Apollo (University of Cambridge) · doi:10.17863/cam.49177 · Cited by 1
Evaluating German Transformer Language Models with Syntactic Agreement TestsPreprint
Zaczynska, K., Feldhus, N., Schwarzenberg, R., Gabryszak, A., & Möller, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.03765 · Cited by 6
Intrinsic Knowledge Evaluation on Chinese Language ModelsPreprint
Wang, Z., & Hu, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14277 · Cited by 1
Disentangling semantics in language through VAEs and a certain architectural choicePreprint
Felhi, G., Roux, J. L., & Seddah, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13031 · Cited by 1
Evaluating Document Coherence ModellingPreprint
Shen, A., Mistica, M., Salehi, B., Li, H., Baldwin, T., & Qi, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.10133 · Cited by 1
Maze Made Easy: Better and easier measures of incremental processing difficultyPreprint
Boyce, V., Futrell, R., & Lévy, R. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/b7nqd · Cited by 1
Language Models Use Monotonicity to Assess NPI Licensing
Jumelet, J., Denić, M., Szymanik, J., Hupkes, D., & Steinert‐Threlkeld, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.439 · Cited by 21
Recurrent babbling: evaluating the acquisition of grammar from limited input data
Pannitto, L., & Herbelot, A. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.13 · Cited by 12
Finding Syntactic Representations in Neural StacksPreprint
Merrill, W., Khazan, L., Amsel, N., Hao, Y., Mendelsohn, S., & Frank, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01594
A Unifying Framework of Bilinear LSTMsPreprint
Rajpal, M., & Low, B. K. H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.10294
Neural network learning of the Russian genitive of negation: optionality and structure sensitivity
Talmina, N., & Linzen, T. (2020) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/z7np-fx81
Self-citationLearning to Evaluate Neural Language Models
O’Neill, J., & Bollegala, D. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-6168-9_11 · Cited by 1
diagNNose: A Library for Neural Activation AnalysisPreprint
Jumelet, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06819
ABSTRACTING INFLUENCE PATHS FOR EXPLAINING (CONTEXTUALIZATION OF) BERT MODELSPreprint
Lu, K., Wang, Z., Mardziel, P., & Datta, A. (2021) · arXiv (Cornell University)
Language Modelling as a Multi-Task Problem
Weber, L., Jumelet, J., Bruni, E., & Hupkes, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.176 · Cited by 8
Is c-command Machine-learnable?
Shin, U., & Song, S. (2021)
Fine-grained Interpretation and Causation Analysis in Deep NLP ModelsPreprint
Sajjad, H., Kokhlikyan, N., Dalvi, F., & Durrani, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08039
Do Grammatical Error Correction Models Realize Grammatical Generalization?
Mita, M. (2021) · Journal of Natural Language Processing · doi:10.5715/jnlp.28.1331
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky ProbingPreprint
Maudslay, R. H., & Cotterell, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02559 · Cited by 6
Towards Zero-shot Language ModelingPreprint
Ponti, E. M., Vulić, I., Cotterell, R., Reichart, R., & Korhonen, A. (2019) · arXiv (Cornell University) · doi:10.17863/cam.43733
Structural Guidance for Transformer Language ModelsPreprint
Qian, P., Naseem, T., Lévy, R., & Astudillo, R. F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00104 · Cited by 1
On the Limits of Minimal Pairs in Contrastive Evaluation
Vamvas, J., & Sennrich, R. (2021) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-206607
On the Limits of Minimal Pairs in Contrastive Evaluation
Vamvas, J., & Sennrich, R. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.5
Metadata
- Authors
- Marvin, R., & Linzen, T.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Linzen, Tal
- ORCID
- Linzen, Tal
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1192-1202
- OpenAlex
- W2888922637
- Cited by
- 334
- References
- 46
- Citations by year
- 2026: 10 · 2025: 15 · 2024: 21 · 2023: 51 · 2022: 38 · 2021: 77 · 2020: 69 · 2019: 51 · 2018: 2
- FWCI
- 20.41
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- grammaticality judgment, subject-verb agreement, negative polarity items, LSTM language models, human vs machine performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
